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The following is a compilation of the presentations made at the 
July 19-23, 1970 National Exemplary Institute in. Squaw Valley/ Cali for” 
nia. The conference was designed to serve as a guide for exemplary 
projects being mdertaken across the country. 

This booklet should serve as a valuable resource for those who are 
involved in the many faceted activities of implementing an exemplary 
program within their state, Many suggestions are offered. Many con- 
cepts and ideas can be used. It is our hope that the time and effort 
of the contributors to this resource booklet will not have been spent 
in vain, but that the information will be used where and when it will 
serve the best efforts of exemplary project staff personnel across the 
country . 



J. Clark Davis 

National Exemplary Institute 

Director 
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Candlelight Planning for a Sateujte World 



Dr* James D. MacConnell 

Professor of Education 
Director, School Planning Laboratory 
School of Education 
Stanford University 



Few of us are capable of living up to the expectations of people concerned 
with education today. Sometimes I feel a sense of kinship to William Howard 
Taft's great grand- daughter who recently wrote an autobiography for her third 
grade class which went like this: "My great grandfather was a United States 

Senator, my father was an Ambassador and I am a Brownie." 

The major problem most of us are facing today is our desire to cling to the 
past, although our culture is a composite of the past. It is the assimilation 
of history, art and science as passed on to us through the centuries. Each of 
the objects around us has a point of origin in the past, but we have no compa- 
rable link with the future. Instead, our concept of the future is a concept of 
continual change. And there is no business or institution which depends more 
ca this concept than education. Businesses are used to planning ahead for at 
least ten year's. But even ten years after our students begin school, most of 
them will still be in school. In fact, most of them will be in some kind of 
structured educational program most of their lives. As we plan for an additional 
one million people in this country by the turn of the century, we must take time 
out to do some chinking. Research shows that if careful planning can guide our 
rapid movements, we will profit both socially and economically. 

Our society is not only changing, but it is changing at a faster rate than 
ever before. Radical change, as exhibited by the growth in population and 
scientific knowledge, is penetrating deeply into the fundamentals of nature and 
life itself, opening fascinating new worlds and creating new problems. Problems 
in planning for the modern society, especially in such far-reaching institutions 
as education, are rooted in tradition and deep emotional feelings. A few com- 
parisons of changing events during our short time on this earth are sufficient 
to jolt the most conservative. Just think — a Boeing 707 or DC-8 taking off on 
an overseas flight burns more fuel in the first three minutes than Lindbergh's 
plane did in twenty-two hours. The country's biggest airline proudly boasted in 
1940 that it would fly 100,000 passengers during that year — the same airline 
is currently carrying that many passengers every two days. In early 1970 when 
the 747 came into service, these data were replaced by even more dramatic figures. 
We associate air carriers with planned progress, yet they too often find their 
long range plans ending up as inadequate short range plans. 
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Planning for the future depends largely upon cooperative activities effec- 
tively organized. We can no longer be loners, but rather, we must be team players 
who can work intelligently with many others on extremely delicate decisions usu- 
ally not under our total control* The question is not whether, but how to plan. 
Who decides, by what means, and for what purpose? What criteria of judgment are 
used? What power is used to persuade or enforce conformity with them? There 
are many problems to be solved and a long way to go before education will have 
reached the levels of modernization and productivity already achieved by many 
other parts of our economic and social environment. 

Being aware of the trends enables us to better plan for the future. There 
will be more older people. Women in the labor force will increase by an hair- 
million a year and will require special kinds of education to assist them in 
re-entering the labor market. 

Still another problem we face is the occupational outlook. Until fairly 
recently, people trained for a given profession were expected to stay at it 
throughout their working lives. Yet, within the life span of our generation, 
the notion of serving in a single occupation for a lifetime is antiquated. Some 
70 percent of the skilled trades in American manufacturing in the year 1900 do 
not exist today, and a large portion of today's skills will become obsolete 
within a very short time. Occupational needs are changing so rapidly that in- 
dividuals must look forward to three or four intervals of retraining during 
their careers, in order to keep up with their professions and trades. Although 
in the past much of this retraining and re-educating has been sponsored and man- 
aged by industry, today the educational institutions of this country are being 
asked to play a key role in this area. The counselor of today must be able to 
project trends in employment opportunities of the future. Compare the employ- 
ment needs of today with those of twenty-five years ago when we had no jets, 
no television or digital computers ; yet today over a million of us are working 
in these fields. And these three areas alone contribute over fifteen billions 
of dollars to our gross national product annually. Until a few years ago 
hard work was lauded — today speed, accuracy and efficiency are winning the 
brownie points. Nineteen hundred sixty-seven was the first year in our history 
when life were paid more for thinking than lifting. 

The population shift to metropolitan areas continues. In 1790 our first 
U. S. census showed that 95 percent of us lived in rural areas. And, as re- 
cently as 75 years ago, two out of three Americans lived in the country. But 
in 1960, more than 63 percent of the population lived in the greater metropol- 
itan areas. In looking at the projected popi/ -.tion growth for the next five 
years , we find that 50 percent of the growth will occur in approximately twenty 
metropolitan areas. This movement is not peculiar to the United States. Popu- 
lation shifts of this magnitude have important consequences for education. More 
people are being educated in the better quality program demanded in metropolitan 
areas. In order to accommodate the migidting population, new facilities must be 
created for demands far beyond those based upon population growth alone. 
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It is evident that workers will work fewer hours, will have longer vacations 
and retire earlier. The average work week has been shortened by eight hours 
since 1945* In California, adult enrollment is increasing at about 50,000 stu- 
dents a year. Almost two million adults are now enrolled in California's schools. 
Because of Federal aid, more attention is being given to basic adult education 
and making the functionally illiterate economically self-sufficient. 

By 1975 unskilled and semi-skilled jobs will decrease from 24 to 20 percent 
(there is no room at the bottom) while managerial and professional jobs will in- 
crease from 22 to 25 percent. Vast adult education programs will be required for 
the next tv/o or three decades to raise the general educational level of workers 
and train them for new skills - new technological changes will make it necessary 
for industry and education to work together on frequent retraining programs, 

More people will be required in service and professional work - jobs requiring 
technical abilities, math, science and personnel relations. This means changing 
curriculum . 

Although the three R's still play a key role today, according to the 1960 
census over twenty-three million persons over the age of 25 had less than eight 
years of school. However, the poor reader has many aids to help keep him in- 
formed if he so desires. Color T.V., Look and Life magazines are making their 
contribution . 

Then, too, the fear of early retirement in an age of inflation is a new factor 
in the worker's already complex life. He has witnessed the disappearance of many 
major occupations of his youth. He now faces the need to learn a new trade or ex- 
perience forced retirement. Both of these problems bring great responsibility to 
those concerned with vocational education. 

New instructional methods are evolving. Enrollment growth, the larger con- 
sideration for students as individuals, new technology and retraining demands have 
made it .lecessary to make changes in the conventional methods of teaching. There 
must be total involvement by teachers and students in the educational process. 

Areas for learning are no longer mere repositories and dispensers of knowledge; 
education is a laboratory for problem solving; it is no longer a telling and lis- 
tening process . 

There is and has been for some time a heavy emphasis on researching the tech- 
nical and ''hardware" aspects of teaching. Audio-visual aids, computers, programmed 
texts, flexible time scheduling, open space architectural designs, etc., have been 
explored and developed to a rather sophisticated degree. The curriculum content, 
the presentation techniques and the physical environment are all under scrutiny 
from diverse points of view, and progress in their design and refinement is certainly 
being made. Our experience with the implementation of such developments, however, 
suggests that no matter what innovative materials or systems to facilitate learning 
a teacher and student may have at their disposal, if they themselves have not al- 
tered some of their most fundamental habits of teaching-learning behavior, improve- 
ment in learning is not realized. In other words, great amounts of money and time 
are being wasted because people have not capitalized on increased opportunities for 
flexibility in behavior. 
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Too much time in structured learning situations is taken up with strangers 
doing things to strangers with the consequence that significant communication 
between instructors and students does not take place. It must finally be re- 
alized that the process of learning, the experience of learning, is part of the 
content of learning. 

Often an attitude of "not wanting to learn" is common for the reason that 
instructors seldom discover how the learner perceives the learning context of 
the educational program and what his own personal ambitions or purposes are. 

He may not want to learn what the instructor thinks he should want to learn at 
a given time. It is appax'ent that many of the acceptable learning and teaching 
techniques for youth today have not been transferred to the areas of adult edu- 
cation. As educational facilities are being planned to accommodate more individ- 
ual learning situations, the adult program should also be geared to using them. 
Instructors are certainly aware that they are faced with a class, a collection 
of bodies , but the significant differences among the students as individuals 
with individual dispositions, ambitions, preferences, purposes, fears, weak- 
nesses, needs, abilities, etc., are virtually unknown to many instructors and 
taboo in group discussions or projects. 

One must know who is doing the teaching and for what reasons, who is doing 
the learning and for what reasons , what is going to be learned and for what rea- 
sons before significant and meaningful learning can take place. 

We must admit that the study of the learning process begins with people, 
and there is no reason why it can't be as tempting and as much fun as pulling 
the lever in some game of chance. 

Everyone realizes that education costs money, but can we afford not to 
participate in the economic benefits of a sound educational system? Centuries 
ago, Confucius said: "If your plan is for one year, plant rice; for ten years, 

plant trees; for a hundred years, educate men." M. J. Rathbone, former Chairman 
of the Board and chief executive officer of Standard Oil Company (New Jersey), 
in a publication entitled Human Talent: The Great Investment , states: "The most 

important capital that any economy possesses are the skills which people carry 
around in their heads." 

We will see the educational programs , methods and equipment of yesterday 
become obsolete and be discarded in another decade. However, the building 
structures designed to house and facilitate current and future programs will 
remain with us for a long time. It is, therefore, of utmost importance that 
facilities be designed for the maximum amount of flexibility to accommodate un- 
known future programs . 

To appreciate the problems in facilities design, one has only to review 
the changes in education brought about by evolving teaching methods, implied 
by terms such as team teaching; mass instruction; small-group instruction; 
listening centers; learning laboratories; study carrels; electronic classrooms; 
study shell centers; and audio-visual centers. These innovations have reshaped 
the learning environment. Where once a school was a large box filled with equal 
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sized smaller boxes called "classrooms today the school is encompassing great 
zones of space -which instructors share, or divide into subspaces according to 
the number of people, the amount of time, and the nature of the learning acti- 
vity. Educators are clamoring for flexibility of space for auditoriums that 
can be immediately divided into classrooms and lecture studios , and for class- 
rooms which can, on a moment's notice, be converted to larger or smaller spaces. 
All of these organizational and physical changes are being made to better ac- 
commodate a more stimulating learning environment. There is a growing demand 
for informal atmosphere, lounge type spaces and furniture for the learner. 

The time is past when we should tolerate shoe-horning adults into fixed student 
desks . 

If America's attempt to educate everyone to his maximum potential is to 
become a reality, much more thought and study in-depth is needed in the years 
ahead. Students today must not only learn at a faster rate than has been 
acceptable in the past, but they must be stimulated as well as aided by the 
instructor. Opportunity must be made available for them to recognize not only 
the school, but also the total environment surrounding the school as a learning 
laboratory. A society capable of continuous renewal will be one that develops 
to the fullest its human resources, that removes obstacles to individual ful- 
fillment, that emphasizes education, lifelong learning and self-discovery. 

Early in life they must be oriented to the fact that most learning takes 
place away from the school and from those of us who are employed for the spe- 
cific purpose of handing down the culture to the next generation. Margaret 
Mead has said that in a simple society where change is slow, the culture can be 
handed down economically from parent to child. In periods of rapid change, 
everyone must learn from everybody else. 

It is now only a matter of time and the manipulating of a distribution 
system when we will See and feel soil from the moon in our local museum or 
school. To live in the future and to live comfortably and happily, we must 
know what it is that we , as a nation and a world, really want. We must be 
willing to use the full resources at our disposal to work for the objectives 
we want to achieve - be it the conquest of poverty, of ignorance, of hunger, 
or of disease . 

There is growing evidence of the need for welding all education and train- 
ing more closely. In the early days of our educational experience, mastering 
the three R's was the goal. This accomplishment was usually scheduled to be 
completed to some degree of satisfaction over a period of eight years . Since 
then we have been adding to both ends of an eight-year span by initiating. and 
financing nursery and pre-school, as well as high school, and special train- 
ing of every conceivable variety, including vocational education. 

At the beginning, it would have been easier to have planned for an orderly 
education system that would have recognized the dignity and educational needs 
of everyone regardless of color or creed. But the stories of the cow jumping 
over the moon were no more fantastic than the Buck Rogers stories that followed 
a few years later and today the moon landing has become a reality. 
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The Characteristics of this Changing Age 
In our Vocational Education 



Dr. mis M. Morton , Jr. 
President, Central Texas College 



Thank you very much. Dr. Hardwick, ladies and gentlemen, it is my pleasure 
to bring to you today a few simple concepts which carry absolutely no endorsement 
by the National Advisory Council on Vocational Education. In fact, it is highly 
doubtful that many educators would take what I have to say seriously. However, 
regardless of what the educators may think or not think, I have confidence that 
those who know the least about professional education will agree with me. 

Standing in that confident position, I will proceed to read to you my per- 
sonal opinions, and nothing more- Hay I say that the first one up in these pro- 
grams usually plays the role of Boogy Han. By the time A1 Riendeau's part is 
presented, things will begin to look better. In any case, the sub-topic of my 
little presentation is "The Characteristics of this Changing Age in our Voca- 
tional Education." 

The position of this country in world leadership, as well as the economic 
wealth and standard of living which many in this nation enjoy, in many respects 
has been generated by the inventive genius of this country's individual citizens 
and their ability to convert these talents to applied science, which is commonly 
known as technology. Thus, the characteristics of the present day in many re- 
spects resemble many of those of previous eras, with at least one exception. 

This primary exception would seem to be that if this country is to retain its 
position of leadership .. .if this country is to provide the gross national product 
whereby an acceptable standard of living is available to all of its citizens, and 
if we are to continue to progress in solving the problems of space-age science, 
air and water pollution, and other problems dealing with the ecology of our na- 
tion, we must re-examine the outlook of the institutions and educational systems 
which each of us represents. 

To review the accomplishments and progress of various technological, voca- 
tional and occupational education programs of the past and their contribution to 
the present status of this country... to reflect on the sources of funds which 
have supported these programs, and to point out the strengths and weaknesses of 
these programs, however excellent or deficient they are, is tantamount to living 
in the past, in a past that has peeled away like an old label in the last few 
years. To be very candid and quite blunt, none of us in the educational commu- 
nity can afford any of these luxuries. The honeymoon with the American public 
is over. During the past ten years, from the time when the public opinion gen- 
erated by sputnik propelled us toward emphasis on all educational programs, vir- 
tually anything done in the name of education was proper. Now, the level of 



productivity at each of our institutions, the lack of efficiency with which we 
operate, and our indecisiveness which has allowed disruptive activities to in- 
terfere with the educational process at all levels, has had a disenchanting 
effect on the legislatures of our various states, the Congress of the United 
States, and the nation as a whole. 

I would remind you of a presentation by Mr. George Klinkhammer of the Bu- 
reau of Education for the Handicapped in which he told the National Advisory 
Council for Vocational Education that "The problem was not in lack of funds 
for training handicapped persons, but rather lack of coordination." This is 
but one example of the preponderance cf evidence that any objective person 
could gather regarding the lack of involvement in productive programs whereby 
the total resources made available for education and training could be brought 
to bear on the problems of the country. 

What I am saying, ladies and gentlemen, is that there are a number of prob- 
lems in education today, many of which will not be solved by money. We as edu- 
cators may blame public apathy toward education and specifically technical, vo- 
cational and occupational training, inadequate funding of these programs, the 
bureaucratic sag which we have allowed to come into being because of our own 
lack of unity, and a long list of other items as excuses which have hampered us 
in discharging the responsibilities assigned us by our constituents, whether 
they be local, state, or national in nature. 

However, two salient facts still stand out. These are (1) it is very dif- 
ficult to solve a problem of which you are a part, and (2) each time a void in 
training and education has developed, some other source has moved to fill that 
void. Again, to be candid, our "slip is beginning to show," and if we are to 
survive the twentieth century as educators, it is up to us to find the answers. 
In short, it is time to either paint or get off the ladder. 

The public, including business and industry, and now national, state, and 
local governments, is pleading for solutions to problems of applied science 
or technology which are available only through technical, vocational, or occu- 
pational training. Colleges and universities, business and industry in this 
country may generate new and innovative ideas, new and advanced scientific 
break-throughs , theories that will assist us in solving the problems of today's 
society. The applied sciences, or as sane would prefer to call it, "Applica- 
tions Technology," necessary to bridge the gap between theory and implementation 
is Ours to deal with and to solve. Conversion of theory, innovative thinking, 
and innovative scientific advances, and the training of persons to implement 
these systems generated by our graduate institutions, business, and industry 
are OUT problems to solve. Should we fail to find the capability to deal with 
these problems effectively in the next few years, I will assure you that the 
twenty-first century indeed will belong to someone else. 

The answer to this dilemma is not in searching for new adaptations in the 
patterns of administration, but in a complete redesign of the present system, 
in which motivation and competition (must) be prime factors. In addition, a 
complete information system regarding the programs which we administer must be 



available at the local, state, and national level to all aspects of government, 
business and industry, and labor. If we are allowed the privilege of glimpsing 
at the twenty-first century, the state of the art will be involvement, coopera- 
tion, and implementation. If we allow the educational system to withdraw into an 
ivory tower governed by pedantic educators, affected and impressed by their own 
thoughts and deeds, it is quite possible we will not even see 1980 in a role 
of leadership. 

Having defined the problem, I should new turn it over to someone else, 
since as an educator, I am normally classified as an f, idea" person, and, there- 
fore, should not expose myself to the details of the matter, or the possibility 
of getting my own hands dirty. Consequently, one might think this is an appro- 
priate place for this speech to end. 

Realistically, none of us desires this approach, and I am certain that you 
would be the first to agree that the responsibilities and problems at hand should 
be addressed by the educational community and prove to all parties that we are, 
indeed, capable of implementing the techniques we teach. 

What are we talking about when we say "redesign?" Simply stated, it is re- 
orienting the present system with three basic principles as objectives, whether 
the problems faced be addressed at the local, state, or national levels. These 
are : 



(1) A system for technical, vocational and occupational training based 
on performance must be established — a measurable performance. 

(2) Involvement in technical, vocational and occupational activities 
which will generate jobs and economic growth for the geographic 
areas to be served must be an integral part of the system . 

(3) A cooperative program must be established to train and retrain per- 
sonnel in all areas, be they handicapped persons, disadvantaged or 
deprived persons, high school graduates, or college graduates. 

This is our responsibility. Others are trying some aspects of these three 
principles, and even building a delivery system — we already have one (history, 
law, continuity) — all we need is a new motivation. 

As to motivation, we must at all levels introduce new methods of handling 
vocational, technical and occupational training programs in a manner whereby 
minimum costs will be realized. This must include a system of funding based 
on actual production, or students trained. New training techniques utilizing 
computer assisted instruction, communications media and television, just to men- 
tion a few, must allow us to teach more students more effectively while retaining 
a "satisfied customer." The days of adding frills just because they are nice to 
have are gone . 

We must face our competition, or better yet, meet It head OD. Otherwise, 
private business, trade schools and organized labor will, and rightfully should. 
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replace us in the educational system. Competition among institutions and among 
programs must be eliminated except where, as _-i the private enterprise system, 
it tends to produce a more effective and more efficient program. Cooperative 
programs with business and industry which tend to reduce the capital invest- 
ment for a specialized program should be implemented where feasible and prac- 
tical. The educational institutions which are publicly supported have too long 
enjoyed the luxury of a monopolistic position in this industry. It is our job 
to compete more effectively with other training programs offered by the private 
business and labor sectors , whereby training costs are kept to a minimum and 
the public interest is served. 

Information systems must be established whereby we obtain a picture at any 
given time of what we look like. Today’s information systems about these pro- 
grams are in most cases inadequate or non-existent. Such management tools as 
cost per contact hour, teacher-student ratios, and job placement data are un- 
available. Data, both as to students and as to dollar expenditures, must be 
input into automated data retrieval systems, whereby, at the local, state > or 
national levels, the profile of vocational, technical and occupational training 
can be determined at any time. It is essential that the "now status” of these 
training programs be available to business, industry and labor in order that 
the trained product, a productive labor force, its capabilities and availability, 
both in terms of time and geographic area, can be input into the national eco- 
nomy as it is trained. The unforgiveable "no-no" must be to train a student who, 
because of the type of training given or lack of capability on the part of the 
institution training him, cannot be placed in a job. The days of training a 
student, presenting him a certificate, shaking his hand and saying "lots of luck” 
are gone. Through effective utilization of information systems, the type of 
trailing, the extent of training and, if you will, the exact twist that the 
training program should take, must be identified while the student is receiving 
this instruction, thereby making job placement automatic and immediate upon com- 
pletion of the program. Only through effective utilization of a well-devised 
straightforward data information and retrieval system can these objectives be 
accomplished. 

Please do not misconstrue these remarks as being those of one who advocates 
abandoning all aspects of the present system of vocational, technical and occu- 
pational training. What 15 advocated is reorienting the present system, since 
the system of funding , the flow of funds , and state involvement through the State 
and National Advisory Council System is working effectively. Essential in the 
redesigned system is funding based on measurable performance and auditable en- 
rollments based on a uniform reporting system; effective, efficient, and innova- 
tive methods of teaching based on actual contact hour costs; administrative in- 
volvement whereby the requirements of business, industry, labor, and state and 
local governments are met. In addition, an information retrieval system whereby 
we can define the system of vocational, technical and occupational training at 
any time for any geographic area is vitally needed. 

The implementation of redesigning, motivating, competing and adapting our 
programs to an information retrieval system will certainly demand the patience , 
creativeness and understanding of all of us. More than this, however, I am re- 
minded of the comments of the person who was recently appointed to a place of 
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prominence in the organization in which he was employed. After being on the 
job one day, he was asked how he liked his new position. His reply was, "After 
all of the well-wishing, handshaking, congratulations, etc., I find I am con- 
fronted with a task which is composed primarily of some nasty old hard work." 

This, I suggest, is the primary job which will be required of us if we are 
to arrive at the twenty-first century as educators. 

In other words, ladies and gentlemen: 

1. The public is on to us; 

2. More production for their money will be demanded; 

3. A patchwork of administrative alterations will not provide the answer; 

*. Powerful and practical educational forces threaten to consume the re- 
maining vocational flesh on our bones; 

5. Only a fundamental redesign will have survival value; 

6. The customer (students and the public) must be satisfied; 

7. The product must be measurable, both in quality and cost; 

8. A uniform (by state) reporting and auditing system must be established 

9. A contact-hour cost by defined course or project must be established; 

10. An information system must be developed to support the previous items. 

What I have suggested may be controversial, but it is conducive to the art 
of survival! Furthermore, most of these concepts are presently being tested in 
at least one state. .They are based or. st.\te legislation recently passed, and 
the productive institutions are delighted with the system. The proto-type of 
the information system, all the way from micro-wave systems to hardware and 
software packages will be ready for testing this fall. They will be serving 
the public sector by the end of this calendar year. (Hay I say that these 
small advances came from people like yourselves.) 

Ladies and gentlemen, this is the beginning of another very long and in- 
volved subject. My wife reminded me this morning that after 25 minutes of any 
of my speeches, people don't continue to wish me well. So thank you very much. 
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"There comes a time in any learning process that calls for reassessment and 
reinforcement. It calls for new directions in our methods of teaching, new un- 
derstanding in our ways of learning, and for fresh emphasis on our basic research 
- . So spoke the President of the United States in his March 3, 1970 message 
to the Congress on education reform. He suggested that the National Institute 
of Education viould begin "the serious, systematic search for new knowledge needed 
to make educational opportunity truly equal." 

Rarely has education had the opportunity to innovate and to demonstrate prom- 
ising examples and models on a national scale as it has today. This unusual op- 
portunity is being made possible by the Exemplary Programs and Projects section 
of the 1968 Amendments to the Vocational Education Act, otherwise referred to as 
Part D of P. L. 90-576. Its primary purpose in the nation is to encourage and 
support the development of pilot and demonstration projects which are based on 
sound research findings, and which promise to improve educational practice. It 
occurs to me that this part of the Act is especially well suited to meet the chal- 
lenges pcsed by President Nixon in his message on education reform . 

Clearly, the Congress of the United States intended all along that vocational 
education should assume a leadership role in the improvement of education. One 
need only scan the wide array of programs throughout the country designed to pre- 
pare students for the world of work to appreciate the impact of this leadership 
since the passage of the Smith-Hughes Act of 1917. Over the years, however, the 
employment picture in America has changed; we have progressed from a simple to 
an exceedingly complex society during the past 50 years. As Dr. MacConnell 
pointed out last night, much of our work has now become increasingly technical in 
nature, requiring skills and educational preparation heretofore unknown. 

Let us consider for a moment the changes that have come about during the past 
half century. Man has walked on the moon. Transportation over land and water and 
in the air has reached phenomenal speeds. Instant visual and oral communications 
are worldwide with the help of a telestar. Surgeons are replacing worn out human 
body parts much as the mechanic replaces worn out parts in the family car. 



These and other changes have been occur! ng at such an astonishing rate that 
adjustment by individuals has been a problem. But there is a general feeling 
that this adjustment problem has not been bothering educators as much as it should. 
We still hear talk about holding the line and not getting excited about ’’innovation 
and those new fangled ideas in our schools.” Such effort to preserve the status 
quo can be likened to the efforts of the master sergeant in the play No Time for 
Sergeants. He was counseling a new recruit from the hill country: "Look here, 

soldier, the Army is like a lake. On it there are many canoes. You’re in one 
canoe, I'm in another. The colonel is in still another. When you start rocking 
your canoe, it creates waves. Those waves rock all the other canoes on the lake. 
Now stop rocking your damn canoe.” 

Exemplary Programs should probably be designed deliberately to create waves. 

By incorporating the most promising educational research findings into a pilot 
program, and selecting an operational setting not unlike other educational settings 
found in that district, our hope is that somehow the good ideas will find their way 
into the school system. 



Part D - Exemplary Programs and Projects 

During the early sixties, an aroused Congress provided new educational legis- 
lation, among which was the Vocational Education Act of 1963 (P. L. 88-210). While 
this Act made possible many educational improvements in the United States, the 
90th Congress saw fit to include features in the Vocational Education Amendments 
of 1968 which would make education more readily responsive to change. 

At the risk of exposing a personal bias , I want to state that one of the most 
exciting features of the 1968 Act (P. L. 90-576) in my opinion is Part D, Exem- 
plary Programs and Projects. Under Part D, vocational education has been given a 
free hand to design innovative and creative programs which have an impact on the 
entire system of education. That the Congress intended this part of the law to be 
comprehensive, there can be no doubt, as witness the following provisions: 

The scope of Exemplary Programs and Projects can include orientation and 
exploration, development of work habits and attitudes, acquisition of 
job skills, and the improvement of teacher competencies. 

The focus of such programs includes all students, but special emphasis 
must be given to noncollege bound youth and more particularly to youths 
with academic, socioeconomic, or other handicaps. Exemplary Programs 
and Projects can be established at all levels of education — elementary, 
junior high, senior high, and postsecondary — and may be directed to 
both in-school and out-of-school youth. 

The setting may remain within the confines of the school or may extend 
beyond to other community agencies and institutions. 
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The Congressional Record carried much testimony during 1967-63 from a wide 
variety of influential sources , most of which supported the concept that educa- 
tion could ar.d should do a better 'job of providing meaningful school experiences 
for all youth. And as Part D v;as hammered into shape by a deeply committed and 
farsighted Education Committee, the magnitude of this new legislation began to 
unfold. The report by the House Committee on Education and Labor in 1968 reads ) 
in part; 

”... The General Subcommittee on Education has concluded that the follow- 
ing five ideas recommended by the Advisory Council (on Vocational Education) 
deserve serious consideration; 1. any dichotomy between academic educa- 
tion and vocational education is outmoded; 2. developing attitudes, basic 
educational skills and habits arc as important as skill training; 3. pre- 
vocational orientation is necessary to introduce pupils to the world of 
work and provide motivation; 4. meaningful career choices are a legitimate 
concern of vocational education; 5. vocational programs should be develop- 
mental, not terminal, providing maximum options for students to go on to 
college, pursue postsecondary vocational and technical training, or find 
employment . " 

To accomplish this, the Congress authorized & budget calling for $57*5 million 
for Fiscal Year 1970, and $75 million for each of the next two years for Exemplary 
Programs and Projects. But for a variety of reasons, not the least of which was 
otr involvement in a costly war in Southeast Asia and the need for supporting other 
high-priority domestic programs at home, the Congress appropriated $13 million for 
1970, and the agreed upon figure for Fiscal Year 1971 has not yet been determined. 



Section 142(d) - State's Share 

Fifty percent of the funds appropriated under Part D of the Vocational Educa- 
tion Amendments is granted to local education agencies and to private and nonprofit 
agencies by the State Boards of education, while the other fifty percent is admin- 
istered by the Commissioner of Education. The manner in which the states choose 
to develop and support Exemplary Programs with their share of Par-t D funds is 
spelled out in their annual State Plans. A great variety of projects and plans 
have been reported by the states, ranging from fragmentation of state allotments 
into $500 projects for individual teacher in-service programs, which includes 
travel for program visitation, to the lumping together of the entire state alloca- 
tion with the Commissioner's share, making one large Exemplary Program. Whatever 
the arrangement, the choice is left to the State Boards. 



Section 142(c) - Commissioner's Share 

While the intent of the Congress was clearly that part D would provide oppor- 
tunities for innovating and field testing of new and improved ideas and practices, 
it soon became apparent to the Commissioner of Education that the $6.5 million 



Federal share for Fiscal Year 1970 v;ould soon be lost in the shuffle unless guide- 
lines were followed. The Policy Paper AVL-V70-1, dated October 2, 1969 > pin- 
pointed the priorities set by the Office of Education for Exemplary Programs and 
Projects vfhich were supported by 1970 funds. In order to achieve maximum impact, 
the entire $6.5 million allocation was focused on programs or projects which 
combined, in one operational setting, of the following provisions: 

1. Provision for broad occupational orientation at the elementary and 
secondary school levels so as to increase student awareness of the 
range of options open to them in the world of work. 

2. Provision for work experience, cooperative education and similar 
programs, making possible a wide variety of offerings in many occu- 
pational areas. 

3. Provision for students not previously enrolled in vocational programs 
to receive specific training in job entry skills just prior to the 
time they leave the school. (Some of these training programs might 
be very intensive and of short duration.) 

4- Provision for intensive occupational guidance and counseling during 
the last years of school and for initial placement of all students 
at the completion of their schooling. (Placement might be in a job 
or in postsecondary occupational training. Placement should be ac- 
complished in cooperation with appropriate employment services, man- 
power agencies, etc.) 

5. Provision for the grantee or contractor to carry the program on with 
support from regular funding sources after termination of the Federal 
assistance under Part D of P. L. 90-576, since Federal assistance 
under Part D cannot exceed three years. 

The first four provisions call for the development of strategies which will have 
impact upon educational areas generally considered weak in our schools. The fifth 
provision carries with it the hope that a good program will somehow be continued. 



Pilot and Demonstration Branch Status Report 

The major goal for which we strive in the Pilot and Demonstration Branch is 
the development of one Exemplary Project in each of the 56 states and outlying 
areas which will be supported by Section 142(c) funds. While it was hoped that 
the entire Federal allotment could be obligated by July 1970, in order to provide 
"tool- up" time for operation of the project when the regular school year started 
in September, there were several reasons why this did not occur in all states. 

One of the reasons was the late start we got due to delayed appropriations at the 
Federal level. 
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The Pilot and Demons trat ion Branch received 168 project proposals for- pro- 
cessing, returned 58 at. nonfundable after an in-house review, then sent out 110 
to be read and rated by a National Review Board. Each proposal was read and 
rated by no fewer than five people, Based on these ratings, selections of the 
most promising projects were made on the basis of one for each state. As of 
July 16, twenty-nine projects calling for the entire state allocation of the 
Federal share of Fart D had been recommended for funding by the Associate Com- 
missioner for Adult, Vocational, and Technical Education. Several proposals 
are in various stages of revievf and processing, while a few states have yet to 
produce an acceptable proposal. Since the Fiscal Year 1970 funds were "no year 
monies" there was no pressure to develop and fund a project in all states by 
June 30. We are fully aware that the January 1 cutoff date was unfortunately 
short, and the quality of the early proposals reflected the lack of sufficient 
preparation time . 

Kew starts were caused for one of two reasons: (1) All projects submitted 

by the January 1, 1970 deadline from a state were disapproved by the Review Board, 
therefore calling for new projects to be submitted by a new deadline set by the 
Associate Commissioner, or (2) The state happened to be where one of the sixteen 
high priority Model Cities 'was located, and where, by order of the Secretary of 
Health, Education and Welfare, special efforts were directed to generate and de- 
velop an Exemplary Project. 

In its effort to attract top quality proposals, the Pilot and Demonstration 
Branch relied heavily upon each of the nine 0. E. Regional Offices, the Research 
Coordinating Units, and the offices of state directors of vocational education 
in each state. Much of the credit for getting proposals written in a few short 
weeks, which included the Christmas vacation, was due largely to the splendid 
esprit de corps which existed among these educational leaders. 

When the January 1 deadline was officially waived for the sixteen designated 
Model Cities projects, one can well imagine that this did not go unnoticed. There 
was consternation about "rule-changing," and some complained that special favors 
were being accorded certain groups. The flood of letters, telegrams, and tele- 
phone calls from the states, as well as from some congressmen, was ample proof 
that Part D had its own watch dog committee. But let me hasten to say that none 
of these messages were of the nasty variety; they were more in the form of "how 
come" inquiries by concerned leaders. 

While we in the Pilot and Demonstration Branch should confess that we were 
as surprised as you with the order that directed the focus on sixteen high priority 
Model Cities, it has since occurred to us that there are some advantages to such 
an arrangement. Much can be learned, for example, by developing Exemplary Program 
models that join forces with such agencies as: Follow Through; Teacher Corps; 

Talent Search; Higher Education Special Services; Higher Education Comprehensive 
Planning; Adult Basic Education; and the Handicapped Early Childhood. So now in 
at least sixteen states, an Exemplary Project funded under Section 142(c) will 
join forces with several agencies to press forward a massive frontal attack on 
inner-city problems. 
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If we added to the Model Cities effort the H.E.W. input of Health, Social 
Rehabilitation and Welfare, the impact on socially and economically blighted 
cities could indeed be formidable. Such a coordinated effort was recently 
described by Dr. A. Neal Shedd, Urban Education and Community Services Program, 
as "having the capacity to produce an extraordinary voluminous orchestration 
of effort.” We were impressed by that statement too. 

As was pointed out earlier, a wide variety of plans for using the Federal share 
of Part D funds are represented in the 56 states and territories. To date, 
seven have chosen to combine the state’s half with the Commissioner’s share, 
giving the project a broader financial base. The seven are Arkansas, District 
of Columbia, Florida, Georgia, Kansas, Mississippi, and New Jersey. The Exem- 
plary Projects approved for funding, which are also linking up with other agen- 
cies under the aegis of Model Cities, are located in Boston, Pittsburgh, Hunts- 
ville (Alabama), Pikeville (Kentucky), New Orleans, and Honolulu. In general, 
these Exemplary Projects are examples of local efforts to improve educational 
outputs in some way by attempting something that would not otherwise be possible 
without Federal support . 



From among the approved projects to be funded under provisions of Section 
142(c), the Commissioner’s share, the following are briefly described: 

1. Colorado — The Aims Junior College District, formally approved in 
the state as an area vocational school, will provide among the sev- 
eral exemplary components, peer counseling for disadvantaged Mexican- 
American students who are potential or actual dropouts from the 
secondary schools. The plan calls for working with the entire family 
unit, with special efforts to be aimed at working with the father. 
Individual learning packages and intensive tutorial assistance will 
be provided these students. 

2. Massachusetts — The New Urban League of Greater Boston, Inc., will 
implement the Exemplary Program through the use of a Continuing Edu- 
cation Center. Unique among the activities at the Center will be 
the advocate and Black exemplar roles to be played by counselors. 

The focus is on inner-city people, mostly Blacks. Incorporated into 
the design of the Boston project are the development of minority ex- 
emplars, occupational information, attitudinal change, parental in- 
volvement, skill training and task analysis. This Exemplary Program 
reflects an effort on the part of a nonpublic sector group to provide 
realistic innovative action in education to meet a serious need. 

3. Virginia — Called the DILENOWISCO Four I's Project (for Intervention, 
Introduction, Investigation, and Involvement), the applicant agency is 
a consortium of five local school divisions headquartered in Wise, 
Virginia. Located in an area of high unemployment, the program is de- 
signed to intervene in the lives of a selected group of youths by in- 
troducing. them to a broad range of occupational information; by making 



Some Examples of Exemplary Projects 
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it possible for them to several occupational areas, 

they will become jjivolved in actual work and learning experiences. 

The target is largely potential dropouts. 

4 . Nevada The Washoe County School District, with offices in Reno, 
developed an Exemplary Project which introduces new elements of 
vocational education at the elementary, secondary, and postsecondary 
levels and combines them with existing elements to form a smooth, 
sequential program. The new elements are occupational orientation 
at the elementary and junior high levels, and a heavy concentration 
of counseling, and job orientation and placement at the high school 
level. A health occupations curriculum at the senior high school 
level is being tried in this program also. 

5 . Pennsylvania — The Pittsburgh Public Schools will, for the seyenth 
and eighth grade orientation program, utilize the facilities of a 
renovated elementary school. Students will be rotated for career 
orientation and exploration. With a centralized location for occu- 
pational orientation, the Pittsburg School District feels it can 
provide a greater variety of materials and equipment as well as 
keep them current at a more reasonable cost. 

The Exemplary Projects described above will give you an idea of the wide 
range of options permitted and, in fact, encouraged under Part D. Within a few 
months it is anticipated that descriptive materials will be available, in either 
hard copy or microfilm through the ERIC System, on most of these projects. 

In a recent report to the American Vocational Association, Governor Buford 
Ellington of Tennessee said: "More important than citing numbers of students or 

dollars invested is the fact that our educational system stresses quality." Based 
upon the proposals for exemplary funds which we have received from all sections of 
the country during the past several months, I would say that educators are most 
anxious to do just that — improve the quality of education. 



Dissemination and Utilization of Knowledge 

One of the objectives of our mission is to share with other educators some 
of the findings which seem particularly significant for educational improvement. 
Plans for dissemination and utilization of the findings of exemplary programs , 
which are themselves based upon solid research, are being carefully considered at 
the National level by U. S. Office of Education officials. They are aware that 
some ideas can be implemented simply by administrative decisions. But educational 
innovation, in order to be implemented, frequently involves new skills, new atti- 
tudes, careful planning concerning supporting roles, and changes in values. The 
problems implied here are often beyond the capability of the school system’s own 
personnel to handle without extensive outside help. 
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There is nothing new about the statement that a lag does, in fact, exist 
between modern technology and the traditional classroom. How wide this gap 
might be is not so important as what educators are doing to close it . The task 
we face is how to mobilize the accumulated knowledge for use by schools in the 
most effective and strategic way. 

The development of operative mechanisms for linking the findings of Exem- 
plary Programs and Projects to widespread practice in our schools about the coun- 
try has raised a host of questions for us lately. Realizing full well the impor- 
tance of getting off to a good start, we have called on such experts as Dr. Lee 
Burchinal and Mr. Thomas Clemens of the National Center ror Educational Communi- • 
cation to help us map out a plan. Since the directors and coordinators of Exem- 
plary Programs about the country will all be actively participating as senders 
and receivers of information for this plan, there will be more on it later. 

Our studies of dissemination and utilization of knowledge have convinced us 
that change doesn’t just happen in our schools; it must be organized -- by people. 
It must be planned by people who are skilled in all aspects of planning for im- 
provement. Since change usually creates conflict within a school and community, 
the idea of improvement must be sold to many people, often on an individualized 
basis . 

If this new effort known as Exemplary Programs and Projects is to have im- 
pact upon the populations targeted by the Congress when they write Part D of the 
Act, namely the youngsters in elementary, high school, and community colleges, 
and especially the disadvantaged and handicapped, they will require a special 
kind of educational leadership. The one who heads up the elementary programs will 
have been carefully selected — he really should be the kind who can walk on water 
and if he doesn’t know how, he’d better learn! Por the successful director or 
coordinator will have interaction with learners, with teachers, with parents, with 
school boards, and with the leaders of the community. He will identify the kinds 
of activities which promote educational improvement, then go about the task of 
implementing them on a pilot basis . 

As plans are made for bringing about change in a school, the following may 
serve as guideposts: 

1. Get to know the principal or chief administrator of the school. He 
may need to be motivated, but since he is the prime influence on 
attitudes regarding the acceptance or rejection of innovation, he is 
the key agent of change in his school. 

2. Develop a systematic, organized approach to adoption of innovation; 
avoid random adoptions amid fanfare and publicity -- out of considera- 
tion for staff morale. 

3. Develop an information system for providing a continuous flow of new 
ideas and new educational products into the school. Be prepared to 
suggest ways of providing specific training as necessary for updating 
the capabilities of teachers to cope with these new ideas and products. 
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4. Above all, maintain an open mind about educational innovation. One 
instructional method may not necessarily be best for all learners. 

It might also be well to remember that school systems have two character- 
istics regarding changes; (1) the tendency to maintain themselves in balance, 
or in a state of equilibrium, and (2) the tendency by people in the system to 
resist change. While this hesitancy to accept change has been the subject of 
much criticism in the past, let us be reminded of the school’s function as a 
social institution. As a reflection of a comparatively 3table society for over 
two centuries, it has resisted attempts at restructuring and has remained stable. 
This is the way our great school system was designed. But there are times when 
the need for adopting new ideas by schools somehow seems most important — and 
perhaps that time has come. The Exemplary Programs and Projects on a pilot and 
demonstration basis appears to be one of the most promising strategies for bring- 
ing about planned change. Such a program can create a temporary new system which 
will demonstrate new ideas and innovations in education. When educators study 
promising exemplary programs, they frequently become their own change agents! 

I wish to close by reminding you that any discussion of the topic ’’Resistance 
to Change" should stress the importance of good salesmanship — the ability to 
explain matters in understandable terms. 

Major John Lindsay of New York tells about a factory owner with one hundred 
fifty-one employees, who proposed a profit-sharing plan — provided every employee 
signified his approval in writing. One hundred fifty men signed immediately, but 
one refused, balking the entire project. For two weeks the holdout persisted, 
then one day he marched into the boss’s office and declared meekly, "I’ve decided 
to sign." "Good,” said the boss, "but what finally changed your mind?" Explained 
the maverick: "This morning the two huskiest members of the union grabbed me by 

the collar and told me^ ’If you haven’t signed up by 10:30 this morning, we’ll 
break both your arms, both your legs, and we’ll knock out all your teeth.’ Well, 
Boss, nobody had ever bothered to explain the plan to me so clearly before.” 
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This word "accountability" shows all the signs of becoming what Don Davis 
calls an "in" word, supplanting "relevancy." And I expect he's right. I also 
suspect therein lies the danger, because accountability is one of those concepts 
with a rich history in the business sector, and if we in education bastardize it, 

we are going to end up with few alternatives, and we have precious few alterna- 

tives to fool around with* 

I hope to bo serious about accountability and show how it is now being de- 
fined. It (accountability) is presently not anything; rather, it is in the state 
of bosoming . There are several tentative definitions r could give you, including 

my own. Let me try to share with you the flavor of the concept of accountability 

and try to show its implications for exemplary vocational education programs. 

I was impressed with Dr. MacConnell’s talk last night and I think you will 
find that much of what 1 have to say will fit in with some of those pecks that 
he provided for us, both in a humorous and in a sober mood. 

The recent near tragedy in space brought the entire nation a lesson in Amer- 
ican courage and ingenuity* This is only the latest in a long aeries of lessons 
being beamed into our homes from these very ur.usual classrooms. Regardless of 
the nature ol the lessons, he it the effects of weightlessness , or a vision of 
the sheer beauty jf the earth, we are privileged to see both instruction and 
management of a very high order in operation. The level of technology, the allo- 
cation of resources, the skillful bringing together of men, materials, ideas, and 
environments all cxpla'n whv it is possible to hold those in our space effort ac- 
countable for results. In my judgement, this lesson in accountability may yet 
turn out to be the most important contribution of the space effort to us in educa- 
tion. Let me defend that. A growing number of influential people are becoming 
convinced that we can't hold the schools accountable as we hold other important 
agencies accountable, both in public and private sectors. Yet, in his March 3 
education message, President Nixon stated, and I quote, "From these considerations, 
we derive another new concept. Accountability . School administrators and teachers 
alike are responsible for their performance , and it is in their interest as* well as 
in the interest of their pupils that they be held accountable . " I could cite for 
you many other pronouncements, and later I will cite another from the New York 
Teacher's Union. 



29 




I want to tell you that Jack Davis is very persuasive. I really had other 
things in mind for this week, but knowing who you were and what the program was, 

I decided I wanted to be here. 

I could give you other statements regarding accountability like the Presi- 
dent’s, from many quarters. For instance, as I’ve gone around the country, I have 
been impressed by people in general (usually they are State Board members, legis- 
lators, governors, and other people in those ranges) and I find that they will 
usually say, "But of course, teachers and administrators will fight this concept 
(of accountability)." And it is the "of course" that bothers me. I'm not finding 
such teachers and administrators. Instead, I'm finding educators who say "How do 
we go about this?" As a result, I hope that most of my remarks will be in a posi- 
tive direction, giving positive ideas which might be applied in your programs. 

I have chosen a quote to sort of balance the President’s statement. The pre- 
amble to the agreement between the Board of Education of the City of Hew York and 
the City Federation of Teachers for the current period (that is, their two-year 
contract) says, "The Board of Education and the Union recognize that the major 
problem of our school system is the failure to educate all , all our students, and 
the massive academic retardation which exists, especially among minority group stu- 
dents; the Board and the Union therefore agree to join in an effort in cooperation 
with universities , community school boards and parent-teacher organizations to seek 
a solution to this major problem and to develop objective criteria of professional 
accountabili ty. ” 

Mow while they speak of massive academic retardation, I would share with you 
some figures former-commissioner Allen gave us just a few months ago. He indicated 
that while the budget in New York, the per pupil allocation from 1960 to 1967, has 
been doubled, the achievement of the children as measured on standardized reading 
tests has been cut in half. There are all kinds of reasons we can give for this 
kind of phenomenon and incidentally, it is rather typical in many places. 

Even though the Union speaks about academic retardation, I would also like to 
share with you a report that was just filed in this state by Arthur D. Little. This 
report is a $400 thousand study of vocational education. In my judgement, it 
ought to he mandatory reading. They discovered in this state, one of the most ad- 
vanced states in education by any common criteria you wish to employ, that one- 
third of the youngsters in this state can be classified as "flying dutchmen." 

Now who are the "flying dutchmen?” You know who they are. They are the young- 
sters for whom there is no curriculum. Neither the academic teachers nor the 
vocational teachers want these children- They are the children to whom we give 
something called general education, an education for which there is no descrip- 
tion and only professional shame. So even though the preamble talks; about aca- 
demic retardation, it could be supplemented in a lot of ways. 

Many more pronouncements and program activities of the sort of those quoted 
above from key groups and important decision-makers could be added. Examples might 
be the recent developments by state governments in Oregon and Virginia. The State 
of Vi ginia’s Board of Education has requested or authorized or encouraged the use 
of the Title I funds through performance contracts to achieve accountability. The 



State of Oregon, among other things, has employed Ray Osborne, Jr. as Director of 
Educational Audits. 0E0, last week, announced the granting of $5.4 million dollars 
to eighteen centers in our country, from McComb, Mississippi to Anchorage , Alaska, 
from Philadelphia to Athens, Georgia, and to other places all over the country, to 
replicate and to enhance the Texarkana study and to look at other incentives in 
education. I could go on. I could talk about the State of Florida, the State of 
New Jersey, the movements in New York. Clearly, a new educational movement is on 
the way. V»e are entering the age of accountability in education. 

The Commission of the States met this month, July 8, in Denver. The Commission 
of the States is a group consisting of governors, members of the state legis- 
latures, and chief state school officers. The theme of that conference, at 
which the national assessment figures were released, was "Accountability. " 

They have said that their fundamental purpose in the 1970's is to pursue accounta- 
bility. Clearly we are entering an age of accountability. 

What is accountability all about? In my judgement, the call for accountability 
in education is a summons to review and reform the educational system. It is not 
another concept; it is not team teaching or flexible scheduling. It did not come 
from our profession; it came from our clients. I want to give you my definition in 
a moment, as well as one or two others which I think are better. Make no mistake 
about it, accountability is not a nice werd, it's a tough word; and I would be the 
first to admit that I don't want to be accountable. And yet 1 would admit that it 
has to be done. I give you a definition from a black parent in the Bronx who came 
to the Office of Education and I was trying to explain some programs that she, along 
with other parents, was not happy with. After my feeble explanation, she said, "I 
know what accountability is/ if you don't learn my child, I'm going to fire you." 

And the Superintendent of Kentucky* last week at a conference at Stone Mountain in 
Georgia with six southern states pursuing the problem of accountability said, "I 
know what accountability is. It's when you go for money and they ask you what did 
you do with that money." So let us not confuse accountability with evaluation, 
assessment or anything else. It is a public policy declaration by the client, by 
the consumer, by the people for whom we work, for whom we are stewards to account, 
to answer for, in a regular way, our stewardship of those funds devoted to educating 
young people . 

In my judgement though, the concept rests on three foundations. I would like 
to use these foundations as the framework within which to talk about exemplary pro- 
grams. These foundations of accountability have been introduced into projects re- 
lating to Titles VII and VIII. They are now in practice in some eighty-six projects 
around the country having to do with the dropout and bi-lingual problems . In these 
projects, there are eighty-six educational accomplishment auditors on the job. They 
are not all independent, that is to say, they are not all different auditors. Some 
are working on more than one project. It is evident that these principles of ac- 
countability are being pursued in the dropout and bi-lingual programs and that these 
principJ.es regarding accountability are being continued even in my absence. 

These three foundations are: 
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First, demonstrated student learning. The focus is on the product, on the 
outcome , on what kids can do as the result of the expenditure of funds and the 
labor of dedicated people. 

Second, these results must be seen through the eyes of someone who is inde- 
pendent of the people producing them. These results in children must be able to 
be replicated by someone qualified, an outside reviewer, an .independent auditor. 

Third, and finally, these results - student accomplishments - as seen through 
the eyes of someone other than the people producing these accomplishments , must be 
reported to the public. It is a public report, which is to say that we who are 

in education are responsible and accountable to the public. We must begin to de- 

velop data and information which can be understood by the public. 

Hone of this is new to you, but the implementation of such a concept as pro- 
nounced by a Congress or a state legislature or a school board is what accounta- 

bility is all about. I think these pillars can fundamentally alter education and 
vocational education. I would like to indicate some of the more important of these 
alterations . 

In the first place, emphasis will shift from teaching to learning. Research 
literature points up the independence of teaching and learning. There can be 
teaching without learning and learning without teaching. And the teacher who says, 
"How proud I am, I flunked 40%" is apt to be met with the answer "Then who needs 
you?" And the teacher who says, "I have very high standards, nobody gets through 
my class" is apt to be met by the statement, "What are your standards, can someone 
else apply those standards? And if you increase your revenue each y«ar , will your 
standards be such that more and more students will be achieving?" There can be 
teaching without learning, learning without teaching. It is not necessarily true 
that teaching results in learning. There can, of course, be learning as a result 
of teaching, hopefully, and I know places where this occurs. So independent is 
this relationship between teaching and learning, ladies and gentlemen, that I could 
cite literature and research indefinitely. It is so prevalent that it is sometimes 
called the phenomenon of the teaching-learning paradox. This suggests that the 
present and traditional methods of requesting resources as well as the principal 
bases for judging the quality of schools will undergo drastic change. In place of 
equating equality in terms of resources allocated (teachers, space and equipment), 
the criterion will be results (how did the students do, how did you live up to your 
promises in your proposal?). This will lead to a second by-product of accounta- 
bility: a revised educational commitment for the nation. 

In principle, the American educational commitment is that every child should 
have access to an adequate education. This is the familiar, the important, the 
magnificent notion of equal educational opportunity. This commitment has been trans- 
lated into legislation as you know, including the vocational education legislation. 
The dollar has been allocated for the people and the things of education, but when 
a child has filed in to learn, school personnel have often assigned to him a label. 
You know the labels - slow, uncooperative, unmotivated, disadvantaged, culturally 
deprived. What a long litany of failures. Accountability would have none of that 
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and triggers a revised commitment, that every child shall learn. And such a re- 
vision demands a "can do" spirit of enterprise, of willingness to change a system 
which does not work and find one that does . A seeking of causes of failure has 
often been in terms of the system, its personnel, its organization and its tech- 
nology. This new commitment to accountability may come to be called the principle 
of equity of results. Former Commissioner Allen's call for every man's right to 
read clearly reveals this new commitment. 

Another major effect of accountability centers on the technology of instruc- 
tion, and the notion of "better standard practice" in America's school rooms. Now 
technology should not be confused with equipment. Technology refers to what works. 
Technology refers to validated practice, to things that are known to work, and are 
then used. Let me talk a bit about standard practice. I didn't say "standardized 
practice." When I was a young fellow, X like to think not-so-long-ago , my father 
was a medical doctor. Being the son of a doctor is not an unmixed blessing, be- 
cause minor ailments are treated when they sometimes would bo better left alone. 
When I had a sore throat, my father had me in his office, using standard medical 
practice for treating sore throats. "Standard" meaning what you would find if you 
went into the doctor's office anywhere around the nation. That standard practice 
consisted of- the following (i..f it brings back memories, so much the better): he 

would seat you and then he would reach into a jar and bring out a tongue depressor. 
Then he would look inside your throat after making you say "ah" and he would reach 
into another jar and get out a little round stick, another piece of equipment. He 
would then reach into a third jar and get out some cotton and deftly, beautifully, 
wind it (i used to try to do that but I never could get it to stick) around the 
stick. He had a jar of argyrol, a black substance, and he dipped the stick and the 
cotton into the argyrol and, using the tongue depressor, would swab your throat. 

He even had a little specially made dish to capture his handiwork. Now, it made 
sense in his time to swab throats because they had learned then that there were 
germs and they knew about antiseptics. The feet that it didn't have anything to 
do with the sore throat was unknown to him. The fact that in the future such 
practice would border on malpractice was unknown to him. 

Medicine has changed, and it has ways to change its practice. Why is it that 
I can step back into my classroom role twenty-two years later and not miss a step? 
That's back. And where is the concept of standard practice in terms of results? 

How doctors in my father's time did not make a lot of money; they were not high on 
the hog. And their sudden rise in power and fame and monetary advantage (and it 
is awesome) came about as a result of their ability to produce results* Not be- 
cause of the American Medical Association, not because of the heavy professionali- 
zation, but because the public was convinced that going to a doctor paid off in re- 
lief of pain or in curing of disease* There is a lesson for education there, which 
seems to me inescapable. 

I'm going to pick up on my formal remarks in a moment, but I want to quickly 
get into how you can engineer accountability into public education and how you 
can utilize this notion, this public policy declaration, in the work we do. Remem- 
ber that there were three critical elements: demonstrated student accomplishment, 

independently perceived, publicly reported. 
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There are several ways to go about engineering accountability into public 
education. There are bad ways, in my judgement, and there are better ways. A 
bad way is to get up on your hind feet as a speaker or a legislator. This was 
done when I appeared before a committee in this state. One of the members of 
that legislature wanted to immediately punish people for what they were allegedly 
not doing. I am pleased to tell you that this person is in a minority. Host 
people are willing to work objectively in the direction of accountability . 

I think the very first thing we need to talk about is developmental capital. 
And let me also talk for a moment about patterns of funding. Z don’t know if 
you have considered that the Federal Government gets an awful lot of mileage out 
of a little bit of money. The Federal Government contributes about 5% of the 
overall elementary and secondary budget, and yet more fuss is made over the 5% 
than the 95%. In my judgement, it is because the Federal government is using a 
pattern of funding through grants management and categorical aids. 

Now we have developed a whole elaborate technology of grants management. 

Some of you are members of that technology. You are the Federal project writer, 
the man responsible for writing and keeping up with all the changes in laws. The 
trips we make, the lobbying we do as members of professional societies, all of 
this effort and influence has one objective and that is to make that 5% larger. 

In my judgement, the allocation of 2 or 3% of the school systems' money as de- 
velopmental capital, as investments in futures, as risk capital to be managed by 
professional people, would be one of the best ways to achieve accountability. 

There is some evidence of this happening. The Texarkana project made use of de- 
velopmental capital, in this case Title VIII funds. Also, the State of Florida 
has now passed a developmental capital program. As a school superintendent in 
San Mateo, I had a school board that was wise enough to set aside 1% of our opera- 
ting budget as developmental capital. That was one-quarter of a million dollars. 

In addition, every penny we could raise from federal, state and local sources was 
put into an account and for several years we were investing $1 million per year in 
new ventures. We also had an academy of instruction, teachers elected by their 
own peers who used a mode of proof called hearings. They held hearings on these 
projects submitted by any teacher or school principal with a hope to do better 
things that would pay off in results. If you had a developmental capital program, 
then you would look at your exemplary program in terms of money set aside by the 
Federal government to be used to achieve certain results. 

Let me go through the elements of the process. It seems to me that unless 
very early in the vocational .education program we adopt an educational engineering 
notion, we are going to go the way of the Title III. Lots of wonderful acronyms 
will be thought up, lots of wonderful things will happen, lots of things won’t 
happen. And then the program will be in the periphery somewhere. No change and 
no legacy will develop and there will then be no support from the Congress and 
others. And if that happens, the program you have ready to launch will not achieve 
what all of you want it to achieve. 

Let me talk for a moment, then, about adapting and adopting, and installing 
good practice. There are good practices that you know or can design with these 
monies. We ought not to forever re-discover the wheel and re-make it. There are 
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things that work, there are professional colleagues of yours and mine who have 
done fantastic things, but what a tragedy it is to see that there is no spread of 
these effects. Who was it who said that it takes forty years for things to get 
into the mainstream? That won’t work. Accountability must spread in one year or 
less. It won’t work to do things the same old way. 

Now what is this process? It is to locate or design good practice (good 
practice means those things that cause changes in behavior in children and which 
you can demonstrate). And once you find that good practice, it needs to be 
adapted. That's a professional job of the first magnitude. There is a wonderful 
program called Outward Bound. It’s a marvelous program for boys. It takes city 
kids and country kids and gives them a chance to pit themselves against reality. 

Not television and not man’s reality, but nature’s reality. It’s a marvelous pro- 
gram, very effective, and it would have awesome effects on our delinquency problem. 
In the city of Atlanta, the Atlanta school system adapted that program. They did 
not have the kind of money to take the boys to the Colorado mountains; instead, 
they took them to come little hills near them in Northern Georgia. They adapted 
the program and still kept its essence. That’s a professional job. 

But that has nothing to do with adopting. Adopting is another program. When 
you talk about adoption, you arc talking about political and economical problems 
that every superintendent knows about, problems that you in vocational education 
know about. That is a political and social problem. It's a sales job. It's a 
packaging job. The people who are good at adoption, at getting things adopted, 
may not be the same ones who know about adaption. 

Then there is the whole notion of installation. How many good ideas, adapted 
and adopted by the board and the staff, fail because the board and the staff do 
not know even the rudiments of logistical support? I can tell you stories that 
are shameful - of school systems that bought things that weren’t even available, 
prototypes; - of manufacturers who inflicted their prototypes on a school system 
and then couldn’t deliver; - of school people acting like complete, utter babes in 
the woods. I can show you beautiful programs on p'aper and then we can talk with 
teachers who need six weeks leave time to get the necessary equipment to make it 
happen. And yet the program says we teach creativity, we want to get that encoun- 
ter. Nonsense. Accountability, if nothing more, ought to teach professional in- 
tegrity, which is to be honest. It has all those earmarks of a hairshirt. It’s 
a very unwelcome concept and I hate to be the man talking about it. 

In the installation phase, you can begin the process of the turn-key. Unless 
your exemplary projects have a turn-key feature, don’t do them. You will get ex- 
pectations in your system that you cannot deliver, and you will be a failure. You 
will have a nice report that nobody in Washington will read and you won't read. It 
will have all the right jargon which will amount to nothing. What is turn-key? It 
is an installation beyond the reaches of that project and unless you plan now on 
what it is to be if you are a success, you will fail. What if your efforts to get 
Federal money really work? Then what? Oh yes, your board agreed that when they 
got the money — but that's a paper agreement. Good intentions to be sure. What 
are the logistical problems, the training problems? Those have to be addressed in 
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the turn-key. That's why I'm going to talk about performance contracts as a tool 
for an exemplary project, not as the way. And so we need to talk about installing 
and we need to discuss a turn-key operation. We then need to talk about feedback, 
and how we are going to monitor this thing because we all know that everything in 
life obeys the second law of thermodynamics. That's a very, very important prin- 
ciple. In corny terms, everything that gets started winds down. No matter how 
gung-ho you are now, a year from now may be different. And our children are tco 
important to us for us to be off on these binges. How do you keep the system from 
losing its momentum, its energy? There are ways. They are called incentives. 

They are changes in the salary schedule, changes in the reward system. This whole 
contingency-management thing is beginning to show us that you can change these 
things. But you must be clever and you must provide for it in your program design. 

One of the things we discovered in our pursuit of accountability was that we 
must utilize a set of tools. My big worry is that the tools become the end and 
not the means. These tools are not accountability. Accountability is a public 
policy declaration. Tools are instruments by which professional people or others 
can hope to achieve accountability. Here are some examples of tools - they go 
under the usual alphabet: MSG, RFP, performance contracts and bids. 

Let me tell you a little bit about these tools, for they may have some ap- 
plicability. We discovered in trying to administer Title VII and VIII that school 
districts did not have the in-house competence to do fundamental jobs. It has to 
be recognized that we in education don't know everything. We should not try to 
be everything to everyone. We must also recognize that there are technical com- 
petencies which we do not have. There are also things asked of us in education 
that we can't deliver. It is important for us to distinguish between education 
and training. Instead of talking about good guys and bad guys, we had better 
talk about burdens and capabilities. 

I heard a Congressional committee being given information from the secretary's 
office, information which said that teachers tend to come from the lowest one- 
fourth of the academic aptitude sector. What does that mean? That's the old busi- 
ness of good guys and bad guys. That's the same conversation you used to hear, and 
still do, where teachers gather and they talk about this year's vintage of young- 
sters - they’re good and last year's are bad. That type of thinking goes clear 
back to Aristotle: that goodness and badness are inherent in the substance. That’s 
nonsense. It's utter rot and ought to be exposed for what it is. You notice that 
goodness is never defined. 

Let's say you select a good teacher. You know how you select a good teacher. 

A teacher is defined rigorously as a "good teacher" if he is trustworthy, helpful, 
friendly, courteous, kind, cheerful, brave, reverent, and obedient. How much bet- 
ter it would be to talk about teachers in terms of burdens and capabilities. What 
are the burdens in the 1970 's (what are we asked to do?); and what are our compe- 
tencies, our capacities, and our capabilities? How can we acquire those talents 
we have to have? Isn’t this what we should be talking about instead of good guys 
and bad guys? 
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Within that context then, one of the things we need to achieve is the ability 

to describe, as rigorously as we can, the things we are trying to do. The very 

first thing we must decide is "what are we trying to make or do?" What do we want 

young people to be able to do as a result of the expenditure of our money? When 

we describe what we want to do, can somebody else equally intelligent, equally 
adept, but outside ourselves, be called in to do these things? If we set up the 
proper conditions, children will perform to specifications and to standards. You 
can then describe your accomplishments in terms ordinary people will understand. 

Let me tell you about that word "ordinary." Ordinary , these days (thanks to 
the great achievement of the public school system), isn't ordinary. Dr. MacConnell 
said this very beautifully. I was asked two years ago to talk to an entire state 
legislature on accountability and the questions they leveled at me were fantastic. 

I turned to the fellow naxt to me and I said "You have some state legislature". And 
he told me that 90% of them had a college degree. Some of them were attorneys, 
some were engineers, and some were teachers. They had come through our educational 
system. Our system hasn't done a bad job for some kids- -it's those flying dutchmen, 
the other 80%, the group in which you are interested, tnat have been neglected and 
by-passed in some places. Go it seems to me that we have to be able to describe 
what it is we are trying to do. Once we have described it, we've got to then begin 
to talk about professional honesty, the toughest topic of all. 

I am impressed with school publications. I used to read school publications ; 

I don't anymore. They are ail the same. Everything is great. I don't care what 
school system it is, in prin -1- it has only success. The kids are happy, there is 
always a smiling face . Yet 1 have noticed there is a correlation between the de- 
gree of happiness displayed in the journals and the failures in tax and bond issues. 
Somehow, the best systems on paper •' e the ones that don't seem to have public sup- 
port. Whoever was responsible for that approach to public relations needs to go 
home. And if you think I'm kidding, try to go before any committee of Congress with 
the following testimoney, which was very common in '65, '66 and '67; "Give us more 
money and we will give you a quality program." Just try that. I'm not responsible 
for what will happen. 

How do we acquire capability and what are some ways a school system can acquire 
capability? Well, obviously, training and going to conferences are acceptable, but 
rather overused, methods. We have to come up with some other tools and I would like 
to share thsrin with you for they may be useful. 

One tool is referred to as the management support group. You've always used 
consultants, as we always have. This tool, however, involves a special kind of 
function where a school system reaches out for people to help them translate their 
, hopes and dreams into something specific. That's not eas} r . We are now placing 

those specifications in a grocery list. The government calls this list, "RFPs" or 
"Request For Proposals." It's a shopping list. It says that we want these things 
| (on the list) done. This is the beginning of accountability. To have accounta- 

: bility we must first have a clear idea of what we want done. The Texarkana project 

may or may not be a complete success, but it has already been successful in some 

• things. It changed the way we went about talking about what we want to accomplish. 
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A school board and a superintendent employed a management support group to help 
them eliminate the educational deficiencies found in their poor and disadvantaged 
children. Their goal was that at the end of the project these youngsters would 
no longer be behind their appropriate grade levels. This goal was stated as a 
specific objective. The district put this objective in a RFP and put it out to 
bid. The private contractor would be paid only for every student who achieved the 
goal. If the student did not achieve the goal, no payment would be made. (Notice 
that the contractor had better produce what he says he is going to produce , or he 
is not going to get paid.) I don't know if it is going to actually work this way, 
but we find expressed here the temper of a public policy declaration of our time. 
And it will be achieved, if not at Texarkana, then in the 0E0 project; and if not 
there, then in one hundred other places. Because it does make sense. It maker, 
sense r.ot to talk about what we cannot do and talk instead of what we can do. We 
have got to find ways to make things work. 

Those who got the contract in Texarkana agreed to specific conditions for 
those youngsters who were three or more years behind in their reading and arith- 
metic, whose parents made $2,000 per year or less, who were primarily black and 
disadvantaged — the contractor would get $80 for each child if, in 80 hours , that 
child achieved one grade level increase in reading as measured by external stand- 
ardized tests and verified by an outside audit. In this case, the Epic Lab con- 
ducted the outside audit. Furthermore, if they had any child who gained a grade 
level in 60 hours or less, they would be paid $106. If a youngster did not gain 
a grade level in reading in 1G5 hours, the contractor would not be reimbursed for 
that student. Instead, they would have to work with that youngster at their own 
expense until that grade level increase was achieved. This is the zero reject 
plan in action. Also, six months after the end of that project, the school system 
could re-test any student for whom money was paid; and if that student had slipped 
below the level previously achieved, the contractor would return part of the fee 
paid for the student's achievement. Do you wonder that that concept is spreading 
around the country like a plague? What a wonderful goal for all of us in educa- 
tion. I can assure you there are many people who say they can do it. They bid 
on an 0E0 project to do just exactly that — to wipe out deficiencies, to get on 
with the job of building a great country where we do not have scholastic cripples. 
It is up to us to make up our minds once and for all to do the job that needs to 
be done and to find ways to make it happen. This is the promise in accountability. 

I don’t have to tell you how important it is in the field of vocational edu- 
cation to find ways to have people successfully employed, to give them marketable 
skills, to put their training on a "no nonsense" basis. Tne process is simple but 
profound. You must reach out if you need expert help to assist you in translating 
our hopes into demonstrable student competencies. You must then put these desired 
competencies into specifications as though you were designing a bridge, and if your 
own staff cannot do it, you must go somewhere else. 

Let me spend a bit of time now on modes of proof. I’m not talking about 
achievement testing, but rather, modes of proof. What are the modes of proof one 
might use in accountability? In my judgment, if we don’t use modes of proof but 
instead go back to the achievement testing of the standardized sort, we wreck the 
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concept of accountability since eveiything does not need to be measui'ed in that 
form. The eye of accountability lies in the phrase "Modes of Proof.” Recogni- 
tion of an expanded notion regarding the assessment of results is the third major 
effect of accountability on school reform. For too long v/e confused measurement 
of results in education with standardized achievement testing of the paper and 
pencil, normal curve-based variety. If you are limited to this useful but narrow 
means of assessment, the pursuit of accountabi iity would be frightening and even 
possibly destructive. Not everything in education can be or ought to be qualified 
in such e manner. Accountability in education, like accountability in other gov- 
ernmental enterprises, can make use of evidence from a variety of modes of proof. 
Education can make use of all these modes and, in the process, use tools for ac- 
quiring evidence such as video tape and pupil performance in simulated real-life 
situations. You in vocational education have an advantage, since you can see a 
student's behavior or a job. To argue that scientific measurement is limited to 
so-called objective tests is to display in.iorance of the rich field of assessment, 
a limited experience with science , and an inability to foresee the rapid develop- 
ment of creative output instruments and strategies which money and attention can 
promote. 

I can show you an Interesting text sock front 1924 by Mr. Turner of Stanford, 
a man of great stature. Nevertheless* , he had biases appropriate For his time. That 
normal curve you know is not just a ‘.cientific device. It has social implications 
wince certain people of our population haw: always showed up on one «nd or the 
other. Hr. Turner wasn’t satisfied to put individuals on that spectrum. Instead, 

Jie had groups- Doctors were on the nice side, school teachers were about in the 
;.:i'idle on the low 3ide, and other people were slotted in elsewhere. He had a dif- 
ferent conception of society . 

. I went back tc school one night in Fairfax County. I was an associate com- 
missioner (for whatever that’s worth) and the fallow next to me was a Colonel in 
the Pentagon. Another fellow was working as a deputy mediator in the Department 
of labor. V/e -were ail parents who had children in that school. A teacher rose 
to tell uu about trie ESC.K biology program there is Fairfax County. By any cri- 
'.«rion, this is or.e of the better programs in the country. This teacher did not 
tell us about BSCS biology, oho didn’t tell us about ecology, and she didn't tel. I 
as about the blood and sweat that went into the curriculum to make biology exciting 
to xhe student. Instead, she said , "We curve the studepfc!" That just makes me 
red. All over this great country of ours, we are curving the student. Now 
it’s time we stopped curving students. It's malpractice, especially if you curve on 
on groups of 30 or less in a classroom. You should understand that this is just 
an example of the nonsense which in going on in the classroom. 

To argue that scientific measurement is limited to narrow, so-called objective, 
tests is to display both ignorance of the rich field of assessment and an unwilling- 
ness to accept what people can do if they make up their minds to do it. I would re- 
mind you of what you already know. For example, in the Armed Forces, a lot of good 
material is available on the accountability model in the vocational training area. 

The outside review component of accountability is the most vital mode of proof 
because science relies for its very existence on qualified review and replication. 
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I'd like to make some predictions about your programs, and hopefully your 
programs will fit these predictions. Wo are really talking about hopes more 
than about predictions- Hare are some of the changes wc would expect in schools 
in vocational education as a result of the call for accountability. When I say 
"changes we would expect," I’m not being naive. I hope for, rather than expect, 
some of these changes. 

The teaching world will finally change from information giving to directing 
learning. Presently, in a classroom, the only person who is active more than a 
fraction of the time is the teacher who is preparing or giving lectures or talks, 
preparing, grading or reviewing tests and homework, cr using that famous formula 
for assignments. The formula I'm referring to is "A - p/t". , ."p" is the number 
of pages in a textbook and "t" is the useable time. Useable time is the number 
of days in the school year minus snow days, minus days removed because of Strikes 
and minus the days students are in the assembly, if the book is twice as long as 
it should be, you then have twice the problem of coverage. This is all being 
done in the name ox individualized instruction. And we are going to see changes 
ir. the way we go about teaching, because teaching isn't necessarily learning. 

Ladies and gentlemen, we are obsolete if we persist in fighting technology, 
if we think "Sesame Street" is going to go away and if we think that the American 
people didn't learn one whale of a lot when those astronauts landed on the moon. 

If we are obsolete, we are going to lose our jobs because technology can do the 
job of information giving better than you and I can. This could mean, then, that 
we have a wonderful opportunity to do the batter things of teaching — the things 
a great teacher does in opening up a field and finding cut v/hat the students real.’y 
want to do. That's a high order teaching skill. If we persist in trying to be the 
information givers, we will continue to be too expensive. 

I think the school facilities will become more open, less restrictive, less 
group oriented. 

The curriculum will become more relevant. When the emphasis moves from pro- 
cess to result, the whole environment becomes a source for* education. Schools can 
then be held in businesses, homes, or out .in the open. Teachers can be assisted 
by students and adults. Since the criterion is results, the process becomes open 
to a variety of input. This has fantastic implications for vocational education. 

In St. Louis, for example, the students are employed by McGraw-Hill and Famoiife -Barr 
Department Stores, etc. These are students who were dropouts — and now they are 
getting a vocational education right there on the job. In Hollywood the child 
actors had to go to school to comply with state law. Los Angeles city schools for 
years have sent the teachers to the studios and the student actors completed their 
education on location when they were through with their job. 

In the President's message in March of this year, he asked us not to confuse 
school with education. Today, our young people spend more time in front of the 
TV than they spend in school. This might be evidence that we need to build an 
educational system where the school is only a small but vital part of the educa- 
tional process. This means we have to work with parents and with industries. This 
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means everyon.* in an educaivr. . om«.* of uu dev professional educator.-: in that we 
! y nov now *o manage the educational tens of vhic;. the school system is only a 
small, ha*, vital, parr of the learning system. Again, w darv* not confuse school- 
ing with getting an education. Be should know hotter. 

Another area o: ot.i-raoded myths, the tv. suit of an incomplete educational tra- 
dition, aunt be exposed and eliminated. Too much of oar behavior toward children 
in the schools seems to reflect a "can't -do* Philosophy, We seen more intent on 
proving that the bell shaped curve* with its built-in reflection of failure, ought 
to be the synod of education. 

Accountability call?; for a new way ox doing business. As that parent said* 
"If you don't learn my child* I'm going to fire you." That's r.ot pleasant news. 
However, L think /America's educators have string guts, powerf.* muscles* and good 
minds. I do:.'! chink that to.*, years from now we are going to have to admit to 



I rather agree with the great, philosopher, Charley Krown. Hi?: mother says, 
"Charley brown, you are not living *..p to your potential." Along come.-: bucy, and 
<. f course she gives it to him with both narrow, by saying, "Cl*. »rlcy brown, you 
just are not J.'vir.g up to your potential. ” Th°n hie principal says, "Charley 
brown, ycu ar-. just not living up to your potential." And Charley Brown, sadly, 
in bit. «.wn way say?:, "There's just no greater burden than a good potential." 

ladies uni gentlemen, your program ha-j groat potcnrial, if you will infuse 
it with the concern for youngster?: which 1 know you have. However, there is no 
longer room for dedication only. We* need dedicated competence. It ycu are will- 
ing and able to incorporate accountability into your programs, I then think you 
are going to succeed and your success will make a difference. 



failure* 
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As a member of the California Vocational staff, T would like to take this 
belated opportunity to welcome you to California and to Squaw Valley. I see 
most of you have gotten the message that the mods of operation in Squaw Valley 
is a matter of informality ami I think the saying goes, "When in Horne, io as the 
Romans do." 

Dr. Lessenger gave you quite a lot to think about regarding this whole area 
of accountability. On the program it said something about using program managers 
or project managers. I’m taking the usual speaker ’•$ prerogative of changing this 
approach today. I think the majority of you would ;?e disappointed if the speakers 
didn't do this. It seems to be expected. 

I.eon talked about change that must come about and must con.e about rapidly. 

He said wo arc in an era where we are being held accountable for- our actions. 

You are here, interested, vitally interested, in getting exemplary programs off 
the ground. I personally believe that if we are to have the type of change society 
demand a of all education, ve have to change our way of thinking to a degree, and 
our way of looking at our educational programs. For this reason, I. have taken the 
liberty of changing what I am going to talk to you about this afternoon. I thought 
for awhile, after listening to Leon, that I wasn't going to have to say anything, 
that he, instead, w-S going to say everything I planned to say this afternoon. He 
dia a good job of it, too. However, there are some things that X am going to re- 
iterate. 3 a;:-, going to do it as simply as possible. 

Wh.it I am suggesting this afternoon is that you give serious consideration 
to one of the alternatives in looking at exemplary pi'ograms. This alternative re- 
quires developing and implementing exemplary programs with a process called systems 
analysis. Now, that's amazing ; I thought at least half of you would throw up your 
hands and leave immediately. At the stage of our development today, when looking 
at this beast we call systems analysis, many of our people in education , I feel, 
are at the same point in their thinking as we were approximately ten years ago, 
when the word "evaluation" came up. We immediately assumed everyone was after us. 

I think today, as Leon pointed out, there is evidence that a number of people are 
after us. 

Systems analysis is, as I have found, not as frightening as many of our en- 
gineers in industry, in defense, in other areas, try to make us believe. To me, 
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I think the best definition of systems is that it is nothing more than the en- 
forcement of the use of common sense in planning, implementing, and evaluating 
our programs . The successful plan and design for a large complex of systems 
requires a systems approach. This systems approach recognizes all the interac- 
tion necessary to tie a total system together. It recognizes that factoring out 
a part of a problem, and then neglecting the interactions among the sub-systems 
and the elements, increases significantly the probability that a solution to the 
problem design will not be found. In other words, we cannot ignore one part of 
our system or another if we are going to come up with meaningful solutions to 
problems. It requires that the boundary of the system be extended outward as far 
as is needed to determine which interactions are significant to the design of our 
system. To implement the systems approach requires the application of a rational 
methodology. Just as the most characterist ?.c feature of science is its method, 
so is the most dominant feature of systems its method. Systems approach requires 
not only consideration of a system large enough to evaluate these interactions , 
but the utilization of systems analysis methodology. 

Three attributes which characterize the systems approach are: 1 ) it empha- 

sizes the importance of the interaction which ties the system together, 2) it 
utilizes the methodology of systems analysis to plan and design a system or pro- 
gram (when I use the term "systems ," I think in terms of our thinking right now, 
you can substitute in place of "systems" - "programs"), and 3) it applies the 
tools and techniques of systems analysis only where they are applicable. 

How a system is a set of elements which is organized to perform a set of 
designated functions in order to achieve the desired results. But first and 
foremost, if you are going to work in the area of systems, we have to have 
definite stated purpose. When you are talking about exemplary programs, we do 
have a definite stated purpose. The stated purpose is spelled out very clearly 
and distinctly in the enabling legislation. < 

Elements are a set of resources organised to perform a highly inr.Tr d 
sub-set of desired functions. The resources which comprise an element in.: ; 
the personnel, material, facilities, and the information needed to do the job. 
Above all we must remember that a system is embedded .in a larger system. Our 
system of education or our system or exemplary programs is embedded in a larger 
system and in that larger system you have environments which are physical, polit- 
ical, social, economic and technological. And these environments comprise the 
super-system or larger system with which there are very strong inter-relation- 
ships. These environments are the sources of the information that you will mu- 1 
in planning and implementing the programs, but more importantly, these environ- 
ments are the sources of the constraints which will limit your program. 

Systems analysis is the process by which people develop the specifications 
of an optimal system in response to the unfulfilled human needs or desires, ft 
is problem solving which involves a quantitative application of technology in or- 
der to identify and prescribe a solution. And the solution is the model withir. 
which you will operate the set of specifications for production, for installation, 
and for producing an optimum system. 
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The design process for developing this system is a fundamental sequence of 
activities which make up each stage of your analysis. Its input is information 
concerning need, resources and environment, and you will notice how needs, re- 
sources and environments continually keep coming up. Fancier systems analysis 
is undertaken in order to fulfill some set of needs or desires of a human being, 
and it is considered essentia 1 that his needs be identified, described and under- 
stood at the very outset. In other words, after your statement of purpose, the 
next thing you have to do is come up with a statement of needs. What is the need 
that will be served by this exemplary program you are looking at? A general state- 
ment of objectives and a knowledge of needs and desires are not sufficient to pro- 
vide an operational guide for your system designer or your decision-makers because 
in addition %o these three factors, \ r ou have to have a system of functions which 
will represent the fulfillment of needs and objectives. 

For instance, the design criteria must be identified and described in general. 
There are three primary classes of criteria: 1) effectiveness or benefit criteria, 

representing the measures oT need fulfillment. 2) resource criteria, representing 
your cost, and 3) schedule criteria, representing the time that will be required. 
These three criteria are necessary in addition to your needs statement and your 
objectives statement- The relative importance of each type of criteria to the 
achievement of the program must be evaluated. The constraints imposed or. the sys- 
tem by its environment must be identified and described. 

Ir. describing these activities, I am certain 1 have confused you to the ut- 
most. Nevertheless, the next process requires that we make up what wc call the 
value model. This is the first basic activity in designing a functional model or 
a program model. The value model includes, in addition to the objectives and de- 
sires, a transformation of these general values to an operational system of cri- 
teria suitable for guiding Che subsequent design planning and decision making. 

Cnoe you have developed a value model, a statement of purpose, a statement of need, 
d. jectives, and the criteria for your design, you then move into the working sys- 
tem. I think in looking at our programs, we may be toying with the idea of clas- 
sifying as exemplary, or trying to get funded under exemplary, programs which, if 
designed to use this approach of systems analysis, will be well on their way to- 
ward making changes, changes which are very much needed. The goal of this whole 
approach is to look at the man-made segments of our environment in education as a 
kind of system and to integrate this Systems concept into our own thinking. This 
special way of looking at the world requires a search for specific purposes and 
requires us to relate functions and processes to these purposes. The systems out- 
look will also require us to assess performance continually, and to exercise qual- 
ity control, and then based on quality control, adjust and plan for improvement. 

The systems approach may call for some change in our way of thinking. When 
we discuss education, for example, most of us can give a precise description of 
schools, facilities, books, personnel, curricula, and other such items. However, 
we become much less specific when it cemes to describing what education "does" and 
we are often extremely vague when we attempt to explain what education is "for." 
The systems approach begins by finding an answer to the question of "What is it 
for?" In other words, in looking at an exemplary program, I think one of the first 
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things we should ask ourselves , in deciding whether we should try it, is “What is 
it for?" After we do this, I think we should then ask ourselves, "What does it 
do?" or "What does it purport to do?" And when we look at the projected outcome 
of this, I think the third question we should ask ourselves is, "So what?" So 
what will happen if these outcomes are, in fact, made into reality. Systems think- 
ing requires that we pay more attention to the attainment of purpose and the assess- 
ment of the attainment of purpose of the system. One way to look at a system, and 
keep its purpose in mind, as our central concern, is to utilize the optimum organi- 
zation and resources available in order to assure that we attain or accomplish the 
performance required in the stated purpose. Therefore, this key criterion by which 
the effectiveness or the adequacy of the performance of a system can be evaluated 
has to do with how closely the output satisfies the pur'pose. It gets back to this 
business of evaluation and accountability. Because a system is accountable for the 
use of the resources that have been made available to it, the economy of the system 
is another criteria by which the adequacy of the system can be assessed. The goal 
is to attain systems objectives with the least possible effort and the least expen- 
diture of time and resources. The systems approach w an be characterized by readi- 
ness to accept and even to plan for changes in the system. Any system that is de- 
signed must be designed with one thought in mind. If it is a workable system, it 
must be a system that will accommodate changes as are dictated by the needs of the 
clientele being served. So it must be something that is viable to change. 

There are six major approaches used in transforming what the industrial com- 
plex calls their systems strategy into that which education can use. This is as- 
suming you have a definitely stated purpose and you have identified the needs to 
be served by this system. These functions are: 1) Formulate objectives which 

are clearly stated in measurable terms. 2) Develop a test. That is, develop 
criteria which will measure the degree to which these objectives are obtained. 

Once the objectives are stated, we must then develop a statement of criteria. 3) 

We must analyze the learning task. In other words, we must ask what the specific 
things are that the learner or the recipient of your program must master in order 
to perform at a level which is consistent with the criteria you have established. 

4) We must design the system. In designing the system, we are talking about 
identifying the functions which must be carried out by personnel in order to sat- 
isfy the learning tasks which, in turn, must be accomplished if we are to meet the 
objectives we have stated. And if we meet these objectives, then supposedly we 
will meet the staged purpose of the entire program. We have to determine who or 
what has the best potential to accomplish these functions and we have to decide 
when and where these functions are to be carried out. 5) The design statement 
and the design system should be tried out and tested. The system can be put into 
operation and the performance of the learner. can be measured in terms of the cri- 
terion which have been established to measure our stated purpose. 6} On the basis 
of the output, we will then have some more information necessary for the project 
manager to make the necessary changes (if any are needed) to completely meet these 
objectives . 

I think the area that Dr. Lessenger pointed out in which we are marked.lv in- 
adequate is the area of stated objectives which are measurable and meaningful. We've 
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given little thought to the assessment of learning tasks and the input competence. 

In other words, we must determine the competency of the person who is going to be 
trained. I think we are also guilty of inadequately using the output from the 
different components of our system. This is feedback needed to make changes. Our 
statement of educational objectives set down by our local boards or found in our 
state guidelines are generally stated in broad and glcwing terms Which can be in- 
terpreted in about any direction anyone wishes to interpret them. Our teachers 
are not usually accustomed to defining learning outcomes in operational and meas- 
urable terms. Without clearly identified learning objectives, your selection of 
materi.als , vour selection of methods, your selection of media will be very diffi- 
cult and those .items which are selected will lack a great deal as far as being 
relevant or appropriate. In stating objectives, they shculd specify what the 
learner is expected to be able to do. 1 think too often we have programs in which 
-,-e carry our objectives around on the top of our head. At the end of the program , 
someone, gets in a flap and we have to evaluate and we have to measure. We then 
come up with tests (sometimes we use standardized tests) and as a result, we find 
ourselves measuring student competencies for which they were probably not trained. 

We need to use verbs that denote observable action. We need to indicate the stim- 
ulus that will get the kind of behavior that our objectives specify. «e must spec- 
ify resources tc be used fcy the learner and by thu persons with whom the learner 
will interact. The objectives should specify how well the behavior its expected to 
be performed by identifying the accuracy or the correctness of response and the 
response length or speed or rate or whatever other criteria you might come up with. 
Also, our objectives must specify under what circumstances the learner is expected 
to perform by specifying the physical or situational circumstances and the psycho- 
logical circumstances under which the learner will function. If our objectives are 
formulated in this way, they will be measurable and they can also serve as a basis 
.for the development of a system. 

Let me share with you the results of the systems analysis workshops which were 
conducted for local district and county coordinators and directors in vocational 
education in California last fall. The purpose of our vocational education program 
in California is the same purpose as in other states. It is spelled out quite 
clearly in the preamble of our enabling legislation. Starting with that purpose, 
a statement of needs was developed by an advisory committee working with us in set- 
ting up these workshops. The functions (see diagram) of our operational program 
(developed through systems analysis) are the product of nine different groups at 
three different workshops. We have subsequently identified the functions people 
felt were critical or necessary in order to have an effective program of vocational 
education. This diagram you have before you is a system design. We realize the 
functional analysis is not perfect. We know ix must be changed as the needs change, 
but it has been adopted on a statewide basis for local districts responding to the 
call for district plans. Districts are asked to design their plan on the basis of 
these twelve functions. They are asked for each function, to identify what they 
had been doing during the past school year, and what they are doing during the pres- 
ent school year, for each of these twelve functions. They are then asked to des- 
cribe the activities they have planned for these functions for the coming school 
year. They are asked to project expected outcomes for the activities they have 
planned, and they are asked to identify questions which they feel should be asked to 
determine the effectiveness of their activities relating to each of these functions. 
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We're just getting our feet wet in this area of systems analysis, but we have 
a number of district plans which have come in that are quite good. It allows our 
people to take a systematic look at their programs in terms of the various func- 
tions in which they might be involved. It also allows them to budget cn the basis 
of these functions which are identified. In other words, how many dollars in terms 
of staff time, clerical time, etc., are you putting into the area of population need 
analysis? I feel that when this additional step is implemented, it will be to the 
advantage of our local people, because it will give them one more tool they can use 
in making a decision regarding the program. We surely know we need all the tools 
we can lay our hands cn in making decisions in education. I think for too long we 
have been guilty of pulling the shade, pulling our left ear, blowing our nose, and 
writing down whatever 1 seems to feel right at the time. We have to look at it in an 
organized, systematic way. We have to look at the alternatives and we have to be 
ready to trade off on these alternatives between the areas of effectiveness and cost. 
For a long, long time we have been operating on the basis that better education for 
youngsters requires more money. By and large, we have been treated pretty kindly 
in terms of financing. Now, lo and behold, we see our bond issues going down the 
drain. It is almost the exception to find one that does pass. We find people crit- 
icizing education and ridiculing us for our efforts. I think that we are finding 
out that people can't afford us. 

The ideas I am proposing with this type of system will assist you in looking 
at, and in evaluating, the programs you think are exemplary. With this system, 
you've got another tool with which to make decisions which have to be made. It 
gives you an idea of what the trade-offs are between cost and effectiveness. In 
other words, "How much effectiveness can you offer in terms of the bucks that are 
available?" It gives you a chance to look at the alternatives. It also gives you 
a chance to discover more efficient ways of operating with the limited dollars 
available . 

This is all necessary because I am firmly convinced that the days of the free 
ride for public education are just about terminated. 
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Emerging Substitute Systems of Education 
for 

NiGH School Dropouts and Potential Dropouts 



Dr. Leon Mi near 

Director, Division of Vocational and Technical Education 
BAVTE , U.S.O.E. 



Many high school students are unable to succeed in regular day-school pro- 
grams. This study reviews substitute systems and programs which have evolved 
outside regular day-schoci patterns in response to the needs of school-alienated 
young people and provides examples which may assist school administrators and 
exemplary project managers in developing new programs. These substitute systems 
include special programs in public schools as well as those conducted by outside 
agencies and firms . 



Agencies and Institutions Which Have Developed New Educational 
Systems for Dropouts and Potential Dropouts 

The need for new approaches in educating school-alienated youth has led 
puolic schools end organizations such as community action groups, business and 
industry, state departments of education, and Federal agencies to develop special 
programs outside regular educational structures. j\ow systems are emerging with 
the institutionalization of innovative projects and programs. 

Some public school systems have been making extensive efforts to prevent 
students from dropping out of school or to provide them with good educations once 
they have left regular high schools. Many different approaches have been tried, 
such as the development of educational centers which provide a new learning en- 
vironment for trie dropout and the use of contracts with private firms for public 
instruction. 

Business and industry have found that basic education is often necessary 
before specialized vocational training of any depth is possible. Employers have 
worked with schools, Federal agencies, and other organizations to help develop 
programs for potential dropouts or dropouts whom they will not be able to hire 
without costly remedial education. Business has not only provided representa- 
tives for advisory groups and positions for work-experience programs, but it has 
also sent company personnel to schools and centers to lecture or assist program 
administration. In addition, it has financially supported innovative orojects, 
often in cooperation with other agencies. 
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Federal Assistance : 

Federal agencies have been active in funding projects and administering pro- 
grams to help disadvantaged youth, potential dropouts, and actual dropouts. The 
U. S. Office of Education (HEW) administers Title VIII, ESEA , which funds dropout- 
prevention programs in public school systems. Requirements for allocating voca- 
tional education funds include spending a set percentage on the disadvantaged, who 
are often the most likely to leave school before graduation. 

Manpower development programs under the institutional training portion of the 
Manpower Development and Training Act include basic education and preparatory vo- 
cational education. These are usually provided at centers which also make supple- 
mentary services available. An increasingly important type of manpower development 
program is on-the-job training (OJT), which enables an individual to begin work as 
soon as possible and thus reduces the cost of support and training. This method 
has been successful for persons who are ready to work and who are not considered 
hard-core disadvantaged. The extension of Federal reimbursement to cover costs of 
providing instruction in basic reading and mathematical skills, employability skills 
(such as good grooming) and preparatory vocational education is necessary if OJT is 
to be of value for the hard-core disadvantaged and school dropouts. 

Community Involvement : 

Community action groups have supported Federal, state, and local programs with 
advice and administrative assistance. They have established programs of their own 
where the need was felt, but they tend to work cooperatively with government agen- 
cies, businesses, and churches where possible, drawing on them for funding, person- 
nel, and facilities. Several innovative methods developed by community action group 
have been adopted by regular public schools — one good example of this has been the 
storefront sunools/street academics approach . 



Adult basic education has been an active force in remedying educational defi- 
ciencies for adults, thus enabling them to improve their situations in life. Re- 
cently the minimum age limit for participation in adult basic education programs 
under the Adult Education Act of 1366 was dropped to age 16. It is estimated that 
about one-fifth of the students in adult basic education programs are ages 16-22. 
Youths who have not been able to achieve even basic reading, English language, and 
mathematical skills are now able to take adult basic courses in order to bring 
themselves up to a higher level of competency. Some states have established com- 
prehensive systems of adult education with the goal of making education from grade 
1 through 14 available for anyone in the state, regardless of age. 

Pj-XVA t ^,.,S„cjlQP.b. : 

Another system which has developed in response to educational needs has been 
the private school which will, for a fee, provide education in any subject from 
remedial reading to technical education. One of the more useful methods employed 
in these schools is the accelerated schedule, in which a student can finish a course 
within weeks rather than semesters. 
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Common Features of Substitute Systems 

Certain commonalities occur in programs for students who have not succeeded 
in regular high school programs . The use of educational centers , where many ser- 
vices as well as educational courses are available, has become an important or- 
ganizational technique, particularly in urban areas. 

Exemplary methods found in many programs for dropouts include individualized 
programs of instruction, programmed learning materials, non-graded courses, assess- 
ment by performance or achievement rather than by time spent, job-sheets or job- 
orders, work-experience programs, small teacher-pupil ratios, and close counselor- 
student relations. 

Students who feel alienated from school frequently have information needs 
which can be met by exploratory courses, occupational orientation , "and training in 
employability skills such as job interviewing and work attitudes. These programs 
can be of great value in motivating students to study and learn, since they permit 
the students to see more clearly the relationship between school and work. 

Close coordination between this type of program and the State Employment Ser- 
vice and local business and industry is another common feature. Dropouts frequently 
want and need work or the realistic expectancy of work in order to be motivated to 
develop basic education and preparatory vocational education skills. Representa- 
tives from the State Employment Service and the business world can materially assist 
teachers and counselors in inspiring young people to study and learn, once job op- 
portunities and requirements are clearly defined. 



Current Exemplary Projects and Systems 

The following descriptions of representative exemplary systems and related 
curriculum trends are offered as sources of ideas for project managers. They all 
may be incorporated into regular high school programs. 

1. NAME OF PROJECT: Urban League contract with Department of Labor 

LOCATION: 26 cities 

SOURCE OF FUNDING: MDTA 

ADMINISTRATIVE AGENCY: Urban League 

TARGET GROUP: 7,000 hard-core unemployed 

ORGANIZATION OF PROGRAM: The Urban League subcontracts for on-the-job 

training to its affiliates in the 26 cities. Employers receive up 
to $1,000 per recruit to help pay for supportive services, such as 
remedial education, job coaching, and minor medical care. 

SCHEDULE OF OPERATION: On-the-job training with reimbursement to em- 

ployer of supplementary costs of training. 
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NAME OF PROJECT: Prevocational Training Center 

LOCATION: Bismarck , North Dakota 

SOURCES OF FUNDING: State Department of Education ; MDTA 

ADMINISTRATIVE AGENCY: Bismarck Public Schools 

SPONSORING AGENCIES: North Dakota State Board for Vocational Education 

and the Employment Security Bureau. 

TARGET GROUP : School dropouts, 16 to 21 years of age 

ORGANIZATION OF PROGRAM : Exploratory center in which a student may try 

out twenty different occupational areas before selecting the one 
he will pursue in regular high school ; other areas are available 
through exploratory work-experience with local businessmen. Basic 
education is provided. 

SCHEDULE OF OPERATION: The center is operated like a job situation: 

punch time clock ; 8 a.m. to 5 p.m. day ; coffee breaks: job orders. 

RECRUITMENT: State Employment Office referrals 

EXPERIENCES OFFERED/ CURRICULUM : Basic education, exploratory experiences , 

guidance and counseling. 

Applicants are tested for abilities and interests, then individu- 
alized programs are developed coordinating basic education with 
regular programs which will be pursued in public high schools. 

Programmed learning materials and job orders are basic training 
tech/iiques . Small student-to-staff ratios are maintained . Group 
guidance sessions provide give-and-take interaction with other stu- 
dents and teachers. Attitude sessions provide employability skill 
training. 

Job orders point out concretely to the student the relationship be- 
tween basic education and vocational training and employment. They 
are used for each vocational area which the student wishes to explore, 
reducing the need for an instructor to provide verbal , time-consuming 
directions . 



NAME OF PROJECT: Philadelphia High School Academies 

LOCATION: Philadelphia, Pennsylvania 

SO’JRCES OF FUNDING: HEW grants t Urban Coalition (business and industry); 

Philadelphia School Board 
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ADMINISTRATIVE AGENCIES: Urban Coalition ; Philadelphia School District 

TARGET GROUP : llth and 12 th grade students with no post high school 

goals 

ORGANIZATION OF PROGRAM: Ten academies , each attached to a specific 

inner-city school . Each academy provides vocational education in 
a particular occupational cluster . 

Project teams, composed of representatives from businesses and in- 
dustries in the Urban Coalition are formed around each academy ; a 
project manager is loaned by one of the companies to direct acti- 
vities of the academy . 

SCHEDULE OF OPERATION: Basic education in the regular high school in 

the morning; specialized classroom work in job training and/or work- 
experience in the afternoon, 

RECRUITMENT: Students are selected from Philadelphia public schools on 

bases of interest, recommendation from principal or counselor , and 
an interview by a project team, 

EXPERIENCES OFFERED/CURRICULUM: Vocational education is organized around 

six occupational clusters identified as areas of high manpower needs 
and job opportunities ; employability skills such as job interviewing 
and good grooming taught by businessmen; work-experience for some 
programs . 

Curricula are designed through task analyses of jobs which are 
then turned into curriculum objectives and learning modules. 

Academies include th.' Business Academy, IBM Academy, Academy of 
Applied Electrical Sciences, Aviation S Aerospace Academy, and 
Health Services Academy — • some of which are duplicated to total 
ten operative " schools within schools.” 



4. NAME OF PROJECT: Street Academies 

LOCATION: New York City 

SOURCES OF FUNDING: Urban League; 0E0 grant; Ford Foundation ; businesses; 

cormuni ty agencies 

ADMINISTRATIVE AGENCY; Urban League 
TARGET GROUP: School dropouts 
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ORGANIZATION OF PROGRAM: Storefront schools — Eleven "Street Academies " 

provide basic education ; two "Academies of Transition " provide gen- 
eral education in a more formal: zed structure / ttvo prep schools pre- 
pare students for college entry. A student progresses from street 
academy to prep school on meeting set reading level standards . 

RECRUITMENT : Handled through t iK>rd-of -mouth and by street workers who 

search pool halls , playgrounds, parks, and bars for prospective 
students. 

EXPERIENCES OFFERED/ CURRICULUM : Students attend in dress of their 

choosing. Study programs are individualized ; effort is made to 
include consideration of personal problems and needs in designing 
an instructional program for a student. Communication is in the 
language of the street. 

No student is asked to leave the program for any reason, though he 
may be reassigned to another phase of the program. 

Among the fourteen sponsors from business and industry are American 
Airlines , American Express, Burlington Insurance, Celanese Company, 
Chase Manhattan Bank, IBM, McGraw-Hill, and Time Incorporated. 

5. name OF PROJECT: Center for Vocational Arts 

LOCATION: Norwalk, Connecticut 

SOURCES OF FUNDING: State Department of Education/ Norwalk Board of 

Education 

ADMINISTRATIVE AGENCY: Norwalk Public School System 

TARGET GROUP: School-alienated youths between 15 and 21 years of age 

ORGANIZATION OF PROGRAM: Occupational education center; guidance and 

work-experi ence emphases . 

SCHEDULE OF OPERATION: Students attend classes three hours daily and 

work part-time for four hours . 

RECRUITMENT : School administrators refer students , using specific 

guidelines deveioped to identify potential school dropouts - 

EXPERIENCES OFFERED/ CURRICULUM : Eight occupational areas; basic educa- 

tion integrated with vocational training ; extensive guidance and 
counseling / work-experience. 



Individualized instructional and guidance progreun is developed for 
each student based on his needs, goals, and abilities. Educational 
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specialists in general education subjects aid teacher and counselors 
in developing the Individualized programs. 

Small group instruction and advanced techniques such as programmed 
materials are an integral part of the curricula. Courses are un- 
graded; progress is determined not by time spent on a task , hut by 
occupational competency achieved. 




NAME OF PROJECT: Adult Education System 

LOCATION: North Carolina — in j^>oIic schools, community colleges, 

churches, trailers , public and private buildings 

SOURCES OF FUNDING: State and local educational agencies; Adult Educa- 

tion Act of 1966 

ADMINISTRATIVE AGENCIES : Community colleges and technical institutes 

under State Department of Community Colleges 

TARGET GROUP: Persons needing adult basic education, ages 16 and up 

TYPE OF ORGANIZATION: Division of adult education within the fifty-four 

State community colleges and technical institutes s some full-time 
centers; evening instructional courses; learning laboratories pro- 
viding programmed learning materials for grades 1-4, open day and 
evening; contractual agreements or affiliations with local sclx>ol 
districts, which issue high school equivalency certificates or 
adult high school diplomas. 

SCHEDULE OF OPERATION: Usually evening courses taught by day-school 

teachers; programmed materials in learning laboratories available 
day and evening. 

RECRUITMENT: Media announcements ; word-of-mouth; churches/ community 

organizations such as Welfare, CAP, Labor, Employment service. 

EXPERIENCES OFFERED/CURRICULUM: Sasic education courses in reading and 

mathematics, integrated in some adult learning centers with voca- 
tional subjects; high school subjects to enable high school comple- 
tion; continuing education. 

Classes include use of mini-labs which enable a teacher to incor- 
porate programmed instruction into the course and individualize 
instruction to meet students 1 learning needs . Learning laboratories , 
which cost $5,000 to $6,000, are available day and night with a co- 
ordinator present. They include organized programmed instruction 
which a student may be phased into at any level from grade 1 to 14. 

No specific programs are designed for high school dropouts, but with 
the permission of the local superintendent, youths 16 years of age or 
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older /nay leave their secondary schools and enter adult education 
programs. This enables them to upgrade their basic education to 
the point where they can return and succeed in secondary school , 
or to complete their high school education in the adult program 
(which may be more acceptable to some students) . 



NAME OF PROJECT .* Metropolitan youth Education Centers 

LOCATION: Denver , Colorado 

SOURCES OF FUNDING : 60 percent by city of Denver; 15 percent by 

Jefferson County; 25 percent under Title I, ESEA 

ADMINISTRATIVE AGENCY: Denver Public Schools 

TARGET GROUP: School dropouts ages 16 to 26 

ORGANIZATION OF PROGRAM: Four occupational education centers; credit 

earned is sent back to high school previously attended (centers 
do not grant diplomas) . 

SCHEDULE OF OPERATION: Year-round classes offered day and night to suit 

student's schedule. 

RECRUITMENT: Word-of-mouth referrals by students in the program; street 

workers who search pool halls and bars; lists of dropouts from 
schools; Employment Service referrals . 

EXPERIENCES OFFERED/ CURRICULUM: Basic and vocational education; small 

classes of no more than six to ten students per teacher; some pro- 
grammed instruction; individualized instruction. 

Close coordination is maintained with the state Employment Service, 
which sends a representative twice a week to interview and test stu- 
dents and to bring information regarding job opportunities and re- 
quirements . 



NAME OF PROJECT: Vocational Village 

LOCA TION : Portland, Ore gon 

SOURCES OF FUNDING: Oregon Department of Education; Portland Public 

School System 

ADMINISTRATIVE AGENCY: Portland Public Schools 

TARGET GROUP: School dropouts ages 14 to 22 

ORGANIZATION OF PROGRAM: Accredited vocational high school for dropouts 

with complete open-door policy. 
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SCHEDULE OF OPERATIONS: Regular school day; regular attendance is en- 

couraged; use of time clocks and cards; fall , spring, and summer 
sessions . 

RECRUITMENT: Word-of-mouth; high school counselors ; court counselors; 

Employment Services; cards mailed to all students in Portland 
School District when they are formally dismissed from school . 

EXPERIENCES OFFERED/ CURRICULUM : Basic education, vocational education , 

work-experience, guidance and counseling. An initial performance 
evaluation is made of each student to assist staff in developing 
his program of study. 

Instruction is achievement-oriented . Job sheets are chief teach- 
ing technique used — each describing one task to be achieved , 
how to do it, what behavioral changes will result. Four or five 
job sheets a day would be average performance . 

Transferring from one program to another is facilitated and en- 
couraged. Programs begin when needed and end when the student is 
ready. Students may work for a high school diploma, for general 
education development , or for a certificate of occupational com- 
petency . Work-experience is an integral part of the curriculum — 
students are placed in cooperative programs as soon as possible , 
while continuing to receive supplementary basic and vocational 
education. Vocational training is handled through the job cluster 
approach . 

Students sign agreements with the staff as to what will be expected 
from a particular course and what will be expected of the students. 
A representative body elected by the students participates in staff 
meetings to draw up the agreements which provide criteria for en- 
tering Vocational Village, participating in programs, and leaving. 
Students are not punished in this school — they have the choice 
of following the agreements or leaving. 



9. NAME OF PROJECT: Dorsett Educational Systems contract with Texarkana 

Public Schools 

LOCATION: Texarkana , Arkansas 

j SOURCE OF FUNDING: ESEA, Title VIII 

ADMINISTRATIVE AGENCY: Dorsett Educational Systems 

; TARGET GROUP: Students in grades 1 to 12 

ORGANIZATION OF PROGRAM: Contract for instruction between a public 

i school system and a private company to design and implement a 
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new system of instructing academic skills, if the company does 
not succeed in raising a student* s performance , it wi22 not be 
paid. 
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Objectives center upon dropout prevention / participants are 
presently drawn from grades 7 to 1.2 . Next year students from 
grades 1 to 6 will be included. 

RECRUITMENT : Random selection from lowest quartile of students in 

Texarkana Public School System. Academic scboois are used 
currently ; students from vocational schools will be added in 
the future. 

EXPERIENCES OFFERED/CURRICULUM: Audio-visual programmed instruction 

from Dorse tt materials , and guidance and counseling to deveiop 
.better study b&bits. Each student must reach predetermined 
reading and mathematical ability levels on norm-reference test 
instruments before graduation from the program (and before 
Dorse tt receives payment) . 

Two hundred students are currently participating ; four hundred 
will be the maximum number. 



10. NAME OF PROJECT: AVCO Educational Service Center 

LOCATION: Glasgow, Montana 

SOURCES OF FUNDING: Department of Labor — WIN and MDTA 

ADMINISTRATIVE AGENCY: AVCO subcontract 

TARGET GROUP: Disadvantaged persons on welfare or otherwise in need 

of employment . Current group is 70 percent Indian. 

ORGANIZATION OF PROGRAM: Two programs are in operation at the AVCO 

plant at the former SAC i?ase in Glasgow. The WIN program provides 
training for welfare recipients with families , most of whom are 
female heads -of -households . The MDTA project provides training 
for family heads and supporting services for family members. Living 
quarters and day-care centers are available . 

RECRUITMENT: An AVCO recruitment team works with the State Employment 

Service to locate potential participants. 

EXPERIENCES OFFERED/CURRICULUM: Assessment, occupational orientation , 

basic education, GED certificate, vocational training — institu- 
tionaJ and on-the-job, employability skills. Wives receive home 
economics instruction in family skills. 
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Jobs and work~ezperlence positions are found in the local area 
as well as provided by AVCO, when available. 



11. NAME OF PROJECT: Aetna Life and Casualty Clerk-Typist and Clerk 

Program for High School Dropouts 

LOCATION: Hartford , Connecticut 

SOURCES OF FUNDING: Aetna Life and Casualty Company ; National Alliance 

of Businessmen ; Department of Labor 

ADMINISTRATIVE AGENCY: Aetna Life and Casualty Company, Inc . 

TARGET GROUP : Female high school dropouts, 16 to 65 years of age 

ORGANIZATION OF PROGRAM: Training center in company facilities , run by 

Aetna personnel , preparing trainees for employment with Aetna. 

SCHEDULE OF OPERATION: Ten girls at a time are trained for ten weeks 

in the center. The first two weeks are spent full-time in the 
canter; in the remaining eight weeks, mornings are spent in the 
center and afternoons are spent working in the department where 
the student will be located after training is completed. 

RECRUITMENT : Concentrated Employment Program or State Employment Ser- 

vice. Criteria for acceptance Includes low-level of income , prob- 
lem job history, lay-^ff or sporadic employment and fourth-grade 
reading level. 

EXPERIENCES OFFERED/CURRICULUM: Program provides language, typing, 

machine and mathematical skills. Graduates are at Intermediate 
level of employment — clerks or cier^c- typists . 

Social and employability skills are also Included In training, 
such as office etiguette, anvironjnentai survival, nutrition , 
grooming, and interpersonal relations. 

The program Was started six years ago, using only Aetna funds * 

This year the Department of Laixtr and NA9 have assisted In t he 
funding. 
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PRESENTATION OF PAPER AND MODEL ON EVALUATION OF EXEMPLARY PROJECTS 

By CENTER FOR OCCUPATIOWAL EDUCATION, WORTH CAROL TWA STATE 
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Introduction 



Dr. John Coster 

Presiding Director 
Center for Occupational Education 
Worth Carolina State 



1 am here to represent the Center for Occupational Education at North Carolina 
State. We are the second of two research and development centers for occupational 
education. 

At the present time, we are being supported from cooperative research money in 
view of the fact that Congress did not appropriate money specifically for vocational 
education research. Of course, our mission is to improve occupational education. 

We are concentrating primarily on four areas: 

Postsecondary education; 

The middle grades or Intermediate programs; 

Occupational education in rural areas; and 
Evaluation. 

We have been interested in two aspects of evaluation. One aspect is program evalua- 
tion, the other is project evaluation. This is the evaluation of pilot projects, 
primary projects and developmental projects, as well as other projects of this type. 

A little bit later on, you will hear about one of the projects we have evaluated, 
having to do with, services, training and education in rural areas. What I want 
to do here, in setting the stage for the other two presentations, is to talk about 
what I call "The Role of Evaluation in Producing Plan Change," You have heard 
some of the theoretical and practical bases of the exemplary programs and I shall 
not repeat them here. The whole notion at the present time that is being promulated 
in the exemplary programs basically comes down to three points : 

1) There is concern about change. Sometimes I wonder whether the change is 
for the sake of change or whether there is change toward some viable ultimate goal 
and set of objectives about which we must be thoroughly conscious. Obviously, there 
is some dissatisfaction with what has happened in the past. There is an eternal hope 
that what happens in the future will be better. There is the notion that change must 
take place. What we are really concerned about is planned change and moving toward 
certain specific kinds of objectives. 

2) Exemplary programs must be based on the products of research. There has 
been some research in vocational education. Some of our critics say there has not 
been very much, and what there has been has not been very good. Nevertheless, since 
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the Vocational Education Act of 1963, there has been a considerable amount invested 
in the research effort. The product of this research must be manifested in the ex- 
emplary vocational education programs . The exemplary program becomes an instrument 
for putting research into practice. 

3) There is an emphasis on the development of a new model, a new concept, new 
ways of responding to the needs of society by developing new programs . I would like 
to think of exemplary programs as being prototypes of new car models which are con- 
tinually being constructed by car manufacturers. Some dream futuristic plans for an 
automobile which may exist two or three decades hence*, and part of this can be incor- 
porated into new operational designs each particular year, but not all can be imple- 
mented at one time. At any rate, I think of the fifty-six exemplary programs, which 
will eventually be funded in the fifty-six states and territories, as being proto- 
types of the new models which eventually and hopefully will be incorporated first in 
the local school system, second in the state system, and third, in the national sys- 
tem. It would not, however, necessarily be in that order. 

Your contracts should call for evaluation by an outside team. We are here to 
say something about our experiences in the evaluation process and to comment on how 
outside teams might function. Over the period of a year, during which time we have 
been involved in a somewhat limited basis on project evaluation, we have learned cer- 
tain lessons that bear repeating at this particular time. They are lessons which I 
suppose were somewhat costly to us. It took some time, but we think we have learned 
lessons which are important. First, the evaluator must be part of a team--not some- 
one who sits outside looking in at the on-going process. He is concerned, among 
other things, with being able to help define objectives and with helping in changing 
programs where they need to be changed. I cannot imagine a pilot program being stag- 
nant; instead of stagnation, there should be a certain dynamic element in such a pro- 
gram. Dr. Griessman and Mr. Morgan will say more about this particular lesson. 
Second, there is the need for continuous feedback — not only to program managers, but 
also to other people who might be interested in the program. Third, our measuring 
instruments are not what we would like them to be at this time. Fourth, our measure- 
ments are not necessarily appropriate to the population for which our programs are 
being designed. They are not particularly appropriate for the disadvantaged who have 
difficulty with reading and who are not really test-prone as far as their particular 
sphere of experience is concerned. We have had some rather undesirable experiences 
with instruments which were not particularly appropriate to the target audience. As 
a result, we have been looking at other ways to achieve the correct evaluation re- 
sults. This problem is not ours only— other people in the country have been looking 
at other ways to produce evidence. This, basically, is our role: We must produce 

evidence to determine not only that the program objectives have been met , but we must 
also produce evidence to determine if the program is capable of installation. The 
whole notion of c::ortiplary programs is to find new devices which eventually will find 
their ways into the mainstream of occupational education, creating new programs for 
1975 or 1980. 

We must produce evidence that the programs are economically efficient; we can- 
not rely on risk capital forever. These programs must be incorporated on the basis 
of economic efficiency so that they can be incorporated within the existing level of 
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funds. We also want to produce evidence chat the exemplary programs will bring about 
changes not only in terms of programs, but in terms of individuals. These are some 
of the concerns we have in terms of the overall evaluation of exemplary programs. 

Finally, evaluation must produce information which demonstrates that Investments 
in these programs have brought about changes which are worthwhile. In addition to 
accountability, we must think in terms of responsibility. Each person involved in 
exemplary programs has a responsibility to insure that the investments made in his 
particular project are worthwhile. I hope this has provided you with an introduction 
to some of the issues and problems inherent in the evaluation process. 
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An Approach to Evaluation: 

A Model for Evaluating the North Carolina Exemplary Rtogram 

Hr* Robert L. Morgan 

Center for Occupational Education 
North Carolina State University at Raleigh 



The purpose of this paper is to present a description of the North Carolina 
Exemplary Program, henceforth the Apex Program, and the evaluation model and 
methods involved with the evaluation of this program. While this paper is pri- 
marily concerned with the Apex Program evaluation, many aspects of this approach 
to evaluation may be generalized to other programs. 



The Apex Program 

The lineage of this exemplary program can be traced directly to the general 
thinking about career development manifested in the legislation precipitated by 
the high level of youth unemployment. Under Part D (Exemplary Programs and Pro- 
jects) of the Vocational Education Amendments of 1968 (P. L. 90-576, Section 141 ) , 
Congress defined the purpose of exemplary programs and projects: "to stimulate, 

through Federal financial support, new ways to create a bridge between school 
and earning a living for young people, who are still in school, who have left 
school eithe" Ly graduation or dropping out, or who are in post secondary programs 
of vocational preparation, and to promote cooperation between public education 
and manpower agencies.” 

The /pex Program, with its three-year allocation of approximately $400,000, 
will be funded through the Office of the Commissioner. Venn (Policy Paper AVL- 
V/0-1, 1969) pinpointed the priorities that the Office of Education had set in 
light of the 1968 Amendments: 

1 . Provisions for broad occupational orientation at the elementary and 
secondary school levels so as to increase student awareness of the 
range of options open to them in the world of work. 

2 . Provisions for work experience, cooperative education and similar 
programs, making possible a wide variety of offerings in many occu- 
pational areas. 

3 . Provisions for students not previously enrolled in vocational pro- 
grams to receive specific training in job entry skills just prior 
to the time that they leave the school. (Some of these training 
programs might be very intensive and of short duration.) 
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4 . Provisions for intensive occupational guidance and counseling 
during the last years of school and for initial placement of all 
students at the completion of their schooling. (Placement might 
be in a job or in postsecondary occupational training. Placement 
should be accomplished in cooperation with appropriate employment 
services, manpower agencies, etc.) 

5 . Provisions for the grantee or contractor to carry the program on 
with support from regular funding sources after the termination 
of the Federal assistance under Part D of P. L. 90-576. (Federal 
assistance under Part D cannot exceed three years.) (Policy Paper 
AVL-V70-1, 1969.) 

This policy statement, together with Section 141 (Vocational Education Amend- 
ments of 1968, Part D), guided our efforts in developing the exemplary program. 

During the summer of 1969, the Apex community of Wai* County, North Carolina, 
was selected as the site for exploratory work in the development of a middle 
grades program by the Center for Occupational Education at North Carolina State 
University. This project stimulated the interest of school personnel in imple- 
menting a total comprehensive program in occupational education. The interest 
displayed by school personnel in this area was one of the major factors contribu- 
ting to the selection of the Apex attendance area as the locale for the present 
project. A number of other factors also were considered during the selection 
process. Apex is the most rural community in Wake County. The economic focus 
of this; community is undergoing a rapid transition from a predominately agrarian 
economy toward increased industrialization. Although Apex is located 20 miles 
from Raleigh, the character of the population in the community and problems of 
providing adequate occupational education more closely resemble the typical rural 
communities of North Carolina, ar.d, indeed, of the South, than the larger urban 
areas. The transition period has required a re-examination of the needs for 
occupational education. The community itself cannot absorb the products of the 
school in its immediate labor force. The socio-economic level of the community 
is relatively low. The P er capita income is below the average for Wake County 
and for North Carolina. The Apex attendance area received the largest amount of 
Title I ESEA funds of any school system in the county. Approximately 55 percent 
of the students in the Apex area qualified under Title I support, the highest 
percentage of any Wa\e County attendance area. Obviously, the project focuses 
on an area that is economically depressed. 

There is an equal distribution of whites and blacks in the Apex attendance 
area. The proportion of black youths in the attendance area, 50 percent, is the 
highest for any attendance area in Wake County, and is higher than the proportion 
of the black population in North Carolina. According to Mann (1963) the black 
population is increasing proportionally in the target attendance area. The inte- 
gration plan has been completed for the Apex attendance area. The student body 
in each of the schools will be approximately equally divided between black and 
white students. The school dropout rate is now approximately 40 percent, and 
the academic achievement level in the Apex attendance area is the lowest of any 
of the Wake County attendance areas • 
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The Apex attendance area includes four schools : 

1. The Holly Springs Elementary Schools which includes grades 1-5, 
with an enrollment of 250 students. 

2 . The A. V. Baucum Elementary School, which includes grades 1-3, 
with an enrollment of 250 students. 

3 . The Apex Elementary and Junior High School, which includes grades 
4-8, with an enrollment of 800 students. 

4 . The Apex High School, which includes grades 9-12, with an enroll- 
ment of 600 students. 

The central participants in the project, therefore, are the 1900 students 
in the four Apex schools and the 75 administrators and teachers who operate the 
program as well as the parents and other members of the community. Since there 
are no private schools in the Apex attendance area, the project will have impact 
upon all youth in the area in grades 1-12. 



At the time of formal acceptance of the Apex Program, word was received that 
budgetary provisions should be made for an evaluation plan to be carried out by 
a third party. In the case of the Apex Program, the Center for Occupational 
Education is the third party which has been selected to perform the evaluation 
(we also are to evaluate Georgia's exemplary program.) The requirements of the 
evaluation plan are: 

An evaluation plan will be carried out by a third party for evaluating 

the effectiveness of the program. The plan shall describe the steps by 

which the contractor wills 

A. Determine the extent to which the objectives of the program have 
been accomplished, 

fl. .Determine what factors either enabled or precluded the accomplish- 
ment of these objectives , and 

C. Promote the inclusion of the successful aspects of the program into 
vocational education programs supported with funds other than those 
provided under the contract. (Department of Health, Education and 
Welfare, Office of Education, Contracts and Grants Division, 1970.) 

In the remainder of the paper the evaluation plan will be described and the 
rationale for selecting this particular plan will be discussed. One of our pur- 
poses in the presentation of this paper is that recently many exemplary programs 
have been funded and more will be funded in the near future. Presumably, all of 
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these programs will require evaluation by outside agencies . Although the evalua- 
tion system that will be presented was developed specifically for the Apex Pro- 
gram, certain aspects of the evaluation plan appear to be sufficiently genera- 
lizable to apply to many other exemplary programs. It is hoped that this paper 
may serve as a stepping stone to the development of a general model of evaluation 
which may be applied in all exemplary programs. The utility of such a model is 
apparent since it would allow the evaluation results of the various programs to 
be compared, thereby greatly reducing the complexity of identifying the factors 
that contribute to a successful program, and strengthen the arguments for contin- 
uation and expansion of the successful aspects of each program. 



Evaluation of the Project 

The outcomes of establishing this system of evaluation for the project are 
threefold. First, evaluation at the process level allows one to monitor the system 
and its component parts in order to determine if process objectives are being car- 
ried out by project personnel and to identify departures from specified procedures. 
Second, evaluation at the product level enables the examination of the results of 
the project activities in terms of the physical entities produced and the behav- 
ioral changes produced. Finally, evaluation results at the product and process 
level provide the feedback information upon which decision-makers can base their 
system of updating decisions and, given a set of objectives that are fixed for a 
given time period, the set of requirements provided by the U. S. Office of Educa- 
tion can be met. A model of such an evaluation system has been developed by Coster 
and Morgan (1969, 1970) and with slight modification can be applied to the evalua- 
tion of this project. The following section will delineate the evaluation model 
and later the evaluation procedures will be described. 

T M .flqflfiV- 

From the twin sources of the individual attributes and the needs of society, 
the mission of vocational education is specified by legislation, (Vocational Educa- 
tion Amendments of I960), albeit somewhat by inference. U. S. Office of Education 
policy papers have produced more specific goals for particular programs (Venn, 

1969.) These must be translated into more specific objectives. The specificity 
and nature of the objectives differ with the level of operation, and it is desirable 
to examine a wide range of objectives in order to develop those objectives which 
are most congruent with the goals of this legislation and policy. Once the ob- 
jectives are specified, the operational procedures and resources required to attain 
the objectives may then be determined. The operational procedures and resources 
constitute the technology of education, the combination of human resources, hard- 
ware, and software which are needed in an appropriate mix to ensure the attainment 
of the objectives. Included also in the technology is the know-how by which these 
resources are mixed and applied. The methodology, the emphases, the curriculum, 
and the materials all form part of the technology of the educational process. 
Finally, of concern to project evaluation are the actual outputs, or products, of 
the program. The evaluation model to be employed consists of five principal struc- 
tural components: 
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1 . The goals of the program, which are a manifestation of the combined 
mix of the value, structure of society, and 'the attributes of the in- 
dividual, are manifested in legislative intent modified or adopted 

in accordance with the State plans and local policies. 

2 . The objectives of the program (desired products). . 

3 . The process objectives (desired processes), 

4 . The observed process: 

(a) The operational procedures — i.e., the methods, techniques, 
emphases, and efforts — being utilized to attain the objectives. 

(b) The resources — both materials (including facilities, equipment, 
and materials) and human (including teaching, administrative, 
supervisory, service and special staff) — provided to facilitate 
the attainment of the objectives. 

5 . The actual outputs or products of the program, as defined in terms 
stated in the product objectives of the project. 

The static interrelationship of these components is illustrated in Figure 1. 

Evaluation may be directed toward an appraisal of the processes of a project; 
that is, to an appraisal of the operational procedures and the resources available 
to operate the program and to attain the objectives. Evaluation may be directed 
toward an assessment of the actual outputs or products of the program. Tradi- 
tionally, the major emphasis on evaluation has been on the process evaluation re- 
garding such entities as the training and experience of teachers, the hardware and 
software available for the instructional program, the ratio of guidance counselors 
to student enrollment, and the size of classrooms and shops while the product or 
educational programs is oftentimes overlooked. 

The assessment of the product of vocational education is more difficult to 
perform. Yet the crux of the evaluation problem is the congruence between the 
actual outputs of the program and the product objectives of the program. A prime 
concern of the decision-maker is the extent to which these two entities are in jux- 
taposition. The prime function of an evaluation program is to produce the informa- 
tion necessary to determine the extent to which these "two entities are in accord. 

In order for evaluation to be effective, it should be defined in terms of in- 
formation needs of decision-makers. Decision-makers, therefore, were introduced 
into the model. The complete model is shown in Figure 2. The decision-makers 
have been introduced at two points. First, the decision-maker (the superintendent) 
has been introduced between the goals and objectives in this model to denote an 
administrative function. Essentially, this illustrates that the decision-maker is 
responsible for specifying those objectives congruent with the mission, and har- 
monious with the goals set forth by the legally constituted bodies. Second, the 
project director has been introduced at a point between the objectives and the pro- 
cess, or operational procedures and resources, to denote his implementative function. 
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The function of the project director is to design and facilitate the implementa- 
tion of the strategies for the attainment of the objectives of this project. 

Thus both decisionmakers performing administrative and implementative func- 
tions will be provided with information on both the products and processes of the 
project. This information will be expressed in terms of the degree of attainment 
of stated objectives. In addition, the evaluative function will act as a filter 
for information obtained about other on-going programs that have similar purposes . 
Other information needs may be fulfilled through local inputs within the community, 
and as needed, evaluators may be used to aid in the assessment of the information. 
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The evaluation will proceed at three levels: (l) the process level, (2) the 

qualitative product level, and (3) the quantitative product level. It should be 
noted that each level of evaluation corresponds to a level of the objectives. 

At the process level , evaluation is strikingly similar to a process in test 
construction called "content validity." That is, do experts in the area of in- 
terest view the project activities as being adequate for achieving the stated pro- 
cess objectives. The evaluation function at the process level, then, is to provide 
experts with complete and accurate descriptions of project activities that are re- 
lated to respective process objectives, and to catalogue the judgment of the ex- 
perts as well as their proposed alternatives. The results of this phase of evalua- 
tion will be reported fully to the superintendent, and to the project director and 
his staff. Ccndensed versions of the process evaluations and corresponding product 
evaluations will be published annually as Center for Occupational Education Mono- 
graphs. The final evaluation report will be incorporated into the project final 
report and will also be published by the Center for Occupational Education in full. 
Inputs from other exemplary programs will be assessed as needed, and reported to 
the project personnel and superintendent. 

The product evaluation will consist of comparing the expected results, as 
stated in the product objectives, with observed results. The reporting procedures 
will follow the format described above. The remainder of this section will describe 
how each objective will be assessed, and for the sake of brevity, objective numbers 
that are listed in the product objectives subsection will be used instead of quoting 
the objective. 



The Objectives 



The objectives of the project include both process objectives, which refer to 
programmatic changes and product objectives, which refer to changes in behavior of 
the personnel in the total school system. 



The overall process objective of the exemplary program is to implement and 
demonstrate the feasibility of a comprehensive occupational education program in a 
rural school system which will provide for: 
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1 . The intensification of the counseling-placement function in the 

school system to provide specifically for: 

* 

a. The provision of "realistic information" about the occupa- 
tional environment to each student at a level of complexity 
commensurate with his maturity. 

b. The provision of "realistic information" to each student re- 
garding his capabilities and probabilities for success (in 
given occupations). 

C. The provision of practice in decision-making to each student 
with emphasis on increasing the student's proficiency in making 
"rational" decisions. 

d. The intensification of individual counseling for students :’jn- 
mediately prior to leaving school. 

e. The provision of placement services to insure that each student 
who leaves school will be placed in an entry occupation or in 
further schooling, and to insure an essential continuity be- 
tween school and community. 

2. The introduction of a program in elementary schools designed to 

provide specifically for: 

a. The integration of occupational information with basic educa- 
tional skills and the intensification of exposure to the range 
of occupations within the context of the level of maturity of 
the student. 

b. The development of work habits and realistic attitudes toward 
occupations and work. 

3. The introduction of a program in the junior high school designed to 

provide specifically for: 

a. The integration of occupational education with the academic 
curricula at the middle grade level. 



"Realistic information " refers to the best assessment at the present time, 
including projections on wage earnings, longevity, and working conditions. Of 
course, there is a possibility that this information may not be accurate due to 
the time lags that exist between the gathering and publication of data, and in- 
validity of our measuring instruments, if our instruments were perfect and our 
projections completely accurate and rational decision:' always desired, it would 
not be necessary to burden the student with more than a set of optimal occupa- 
tions for his consideration. However, since this is not the case, each indivi- 
dual should be provided with as many alternatives as is possible in order to 
maximize his probability of occupational proficiency . 
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b. Realistic exposure to the range of occupations in the community, 
state, and nation, including up-to-date information as to know- 
ledge, skill and training requirements and benefits to be accrued 
from training. 

C. Realistic exposure to the knowledge of one's self, including the 
beginning elements of understanding the attribute mix of the 
student . 

d. The introduction of the career decision-making proce s, including 
the choice and consequence of alternatives. 

e. The provision of "hands-on" experience in occupational labora- 
tories, and on-site observation of work. 

f. The provision of appropriate skill training for students who have 
decided to leave school prior to completing junior high school as 
a "vestibule function.” 

4. The expansion of the occupational education programs in the high school 

program to provide specifically for: 

c. The integration of occupational education with the academic pro- 
gram at the secondary level. 

b. The equipping of each secondary school student who does not plan 
to continue formal schooling with a job entry skill. 

C. The intensification of the counseling-placement function to insure 
that each student is prepared to obtain employment in an occupation. 

d. The expansion of opportunities for cooperative education and work- 
study programs. 

G. The provisions for appropriate skill training for students who 
have decided to leave school prior to graduation as a "vestib ul e 
function.” 

The process objectives will be evaluated by expert judgment. The processes 
will be recorded in accordance with these objectives in order that other interested 
parties may evaluate the attainment of the objectives. These objectives will be- 
come more specific as project personnel are employed and the program is established. 



Product Objectives 

The product objectives include qualitative and quantitative manifestations of 
behavior which are expected to change as a result of the proposed project. 
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. Qualitative Objectives 

1. To increase the student’s interest in and awareness of occupations 
in his community. 

2. To increase the student's interest in academic subject matter areas 
by incorporating occupational information into the curriculum. 

3. To increase the interest of parents, teachers, and students in occu- 
pations . 

4. To increase interest in employee abilities and attitudes by students, 
teachers > and parents. 

5. To increase satisfaction with curricular offerings. 

C. To increase interest in posts econdary training. 

7. To increase interest in occupational course offerings at the high 
school level. 

8. To increase interest in obtaining entry level skills. 

9. To increase the student's knowledge of the occupational enviionment 
and his own abilities. 

10. To increase the number of "rational" occupational decisions. 
quapi-|tative Objectives.: 

1. To increase the number of occupations that a student can name by 50 
percent each year. 

2. To have each student know the occupations of each of the members of 
his immediate family (mother, father, brothers and sisters). 

3. To increase the number of "good" work habits that each student knows 
by 50 percent each year. 

4. To increase average academic achievement by 5 percentiles each year, 
as measured by the California Achievement Test. 

5. To increase the average daily attendance percentage by 5 percent each 
year. 

6. To reduce the number of grade failures by 5 percent per year without 
altering academic standards. 

7. To increase the number of parental conferences requested by 5 percent 
per year. 
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8. To decrease the dropout rate by S percent per year. 

9. To increase the number of students in work experience programs by 
10 percent per year. 

10. To increase the number of requests for career guidance services by 

20 percent the first year and 10 percent each of the following years. 

11. To increase to 100 percent, in three years, the percentage of persons 
with job entry skills, who do not plan to enter a postsecondary school. 

12. To increase the number of students using the occupational information 
center to 30 percent of the students enrolled within three years. 

13. Tc place all (l</0 percent) of graduates and dropouts that seek employ- 
ment v/ithin three years. 

14. To increase the proportion of students in the high school enrolled in 
vocational programs by 5 percent per year. 

15. Tc increase the number of course offerings in vocational areas by two 
courses per year. 

16. To increase the number of students ap: lying for postsecondary educa- 
tion by 5 percent per year. 

The evaluation system will be employed to monitor and update the system ob- 
jectives and program performance. Since the evaluation is dynamic, the objectives 
can only be viewed as fixed, prior to the first evaluation, which will be based on 
the performance of the Apex project and other exemplary programs, as well as new 
research findings. 

Qualitative objectives 1, 2, 3, **, 5, 6. 7, and Q will be assessed by de- 
veloping questionnaires based peychometrically on Likert scaling techniques. The 
increases in the various qualities will be evaluated by a pre-test — post-test 
paradigm, as will all other product objectives. Baseline measures will serve as 
the starting point against which measures obtained at a later time will be compared. 

These questionnaires will be designed for persons in various age groups that 
are appropriate for the assessment of given objectives. Consultants will be em- 
ployed to aid in the development of these questionnaires as well as for other as- 
pects of the product evaluation. 

Qualitative objective 9 will entail drawing upon items from tests that have 
previously been developed to measure occupational knowledge, and design a test ap- 
propriate for each age group. The knowledge of individual abilities will be as- 
sessed by comparing self-ratings with best results and ratings by other persons. 

The increase in knowledge would correspond to a greater degree of agreement be- 
tween the self-ratings and the criterion measures. 
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The assessment of "rational decisions" (qualitative product objective 10) 
will be limited to grades 7 through 12. The cox'respondence between stated oc- 
cupation choice, and the probability of success in the chosen occupations, will 
be assessed by staff members. If the probability of success .is rated below .5, 
the occupational choice will be considered "irrational decision." 

The quantitative product objectives, like the qualitative product objec- 
tives, will use baseline measures obtained at the onset of the program as a 
relative zero point. Quantitative objectives 1, 2, 3, 4, S, 6, and 7 will be 
assessed for grades 1-12; objectives 8, 9, 10, 11, 12, and 13 for grades 6-12; 
and objectives 14, 15, and 16 for grades 9-12. 

Objective 1 will be assessed by simply having each student list all of the 
occupations that he knows and if, for example, he listed ten occupations on the 
pre-test, a 50 percent increase would require that he be able to list fifteen 
occupations at the beginning of the second year, twenty at the beginning of the 
third year and twenty-five at the end of the project. 

Objective 2 will be assessed by a listing of occupations of the immediate 
family. The list will be compared with school records. 

Objective 3, like objective 1, will be assessed by simply listing "good" 
work habits. The goodness of the work habits that are listed will be evaluated 
by staff members. A simple frequency count of the "good" habits will be com- 
pared with the baseline measure to ascertain percentage increase. 

Objective 4 will be assessed by using population norm deviations to obtain 
percentiles at the baseline. These norms will also be used to ascertain the 
percentiles from which the baseline measures will be subtracted. 

The average daily attendance at the high school level is approximately 83 
percent. To fulfill objective 5 the attendance percentage must rise to approxi- 
mately 95 percent. 

Without a change in grade policies , the number of grade failures must be re- 
duced by 15 percent of the original number, to fulfill objective 6. A baseline 
measure will be used. 

The number of parental requests for consultation about their child's career 
plan must be increased by 5 percent per year in order to fulfill objective 7. 

The dropout rate in Apex High and Apex Consolidated is approximately 40 per- 
cent. To fulfill objective 8 at the end of the program it must be 25 percent or 
less. 



The fulfillment of objective 9 is contingent upon increases in the proportion 
of students in school supported work experience by 15 percent, compared to base- 
line measures. 
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The fulfilment of objective 10 is based on increasing ‘requests for "career 
guidance services," as defined by focus of the request, by 20 percent over the 
baseline measure for the first year and 20 percent the next two years. 

The fulfillment of objective 11 requires that each person who graduates 
from Apex High School, and does not plan to continue education, will be equipped 
with entry level job skills. 

Objective 12 requires that 80 percent of the students in the middle grades 
must "use" the Occupational Resources Center. Use is defined as spending at least 
one hour per semester at the Center. 

The fulfillment of objective 13 is contingent upon placing each student who 
requests placement in a position within a twenty-five mile radius of Apex, North 
Carolina. 

The percentage of students enrolled in vocational programs is approximately 
25 percent. To fulfill objective 14, the percentage enrolled must increase to 40 
percent. Vocational programs are defined for evaluation purposes as courses that 
provide the student with job entry skills. 

Objective 15 is fulfilled by adding two courses per year to the vocational 
program curriculum for the three year period. 

To fulfill objective 16 it is necessary to increase the proportion of stu- 
dents applying for postsecondary education hy 5 percent per year. Postsecondary 
education means at least one year of education after the completion of high school. 

i As was stated before, the project process and product objectives will be 

reviewed by the decision-makers constantly. Revisions of the objectives are, of 
course, subject to U. S. Office of Education approval. The changes in the objec- 
tives may occur on a semi-annual basis the first two years and an annual basis 
thereafter, 

The plan for evaluation in Apex is not based on a random selection of stu- 
dents since Apex is an intact sample. Each child will be included in the evalua- 
tion.' A control group composed of randomly selected students from schools identi- 
fied by state personnel as being most advanced in vocational education practices 
i and most similar to Apex will be selected. These students will only be assessed 

once at the end of the third year of the Apex program. This method is called a 
post hoc design. 

Before one attempts to develop an evaluation plan, the question of why is 
■ the evaluation being undertaken must necessarily be answered. One answer to this 

( question for the Apex program is assumed to be: evaluation is undertaken in order 

\ that the decision-maker may be provided with information that can be used to im- 

; prove his program. This function corresponds to what Scriven (1967) calls "forma- 

tive evaluation". Formative evaluation is a developmental form of evaluation in 
which the evaluator's responsibility is providing information upon which judge- 
ments about program revisions can be based. Another assumed answer in the Apex 
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program is: evaluation is undertaken in order to determine if a program, as a 

whole, is effective. This corresponds to Schiven’s definition for summative 
evaluation. Summative evaluation seems to be the type of evaluation described 
in the U. S. Office of Education specifications. 

Continuing on this line of reasoning, certain other assumptions have been 
made* The first assumption is that, at best, the Apex program is sophisticated 
guesswork. That is, many aspects of existing research and development projects 
have been combined into a single program. Since the Apex program is unique, one 
cannot say what effect the interaction of the selected factors will have on the 
program. The second assumption is that even if the Apex proposal were essen- 
tially perfect, a certain amount of "slippage" would occur because communication 
systems among humans are somewhat less than perfect. Hence, formative evaluation 
will be undertaken, and the program will be revised, or at least reviewed, semi- 
annually for the first two years of operation. 

It is also assumed that the Apex program must justify its effectiveness to 
the funding agency specifically and to the public in general. This primarily 
involves the identification of products that the program has produced, as differ- 
entiated from those that might be produced without the program. Hence, baseline 
data and a comparison (control) group is necessary. However, the comparison 
group we have selected would be expensive indeed to follow throughout the course 
of the project, therefore we must accept the weaknesses of a post hoc design, 
with no pre-test on the control. The differences in the two groups will be as- 
sumed to be attributable to the program. The summative evaluation is limited to 
the third year of the program and objectives will be fixed during this time period 
The efficiency of the program cannot be determined except by comparison with simi- 
lar programs or indirectly by expert opinion. Almost every evaluation plan, if 
not every plan, has been damned and praised by various sources and for various rea 
sons. We expect little better for our plan. For example, Guba (1969) virtually 
annihilated all current models of summative evaluation. This was followed by 
Light and Smith (1970) who credited current evaluation methods with possessing 
far greater power in detecting failure than inspiring success. Stufflebeam (l97o) 
attacked summative evaluation on similar grounds. Perhaps the most damning of all 
criticism of summative evaluation was produced by Wolf (1969) with his tongue-in- 
cheek "colloquial method." 

Social psychological research has demonstrated that decisions arrived 
at by a group w ill achieve greater acceptance than decisions arrived at 
by an individual. This finding is the basis of the colloquial method. 

In applying this method , one need mer^ y assemble a group of people who 
have been associated with a particulc ~ program to discuss its effective- 
ness. After a brief discussion , the group will usually conclude that the 
program has been indeed successful. This conclusion can then be trans- 
mitted to funding agencies and other school personnel. It is Unlikely 
that such evaluations will be challenged since they have been arrived at 
by a group. 



78 



83 



i 



Formative evaluation, on the other hand, has received relatively little 
criticism. This may be due to the fact that the evaluator serves in an infor- 
mation-gathering capacity and does little in the way of threatening the exis- 
tence of the program. The main question raised is: is this really evaluation? 

The Apex Plan combines both types of evaluation and might be criticized on 
the basis that the decision-maker and evaluator lead symbiotic existences for 
such a long period of time as to preclude objective summative evaluation, yet, 
realizing this weakness, we contend that if the evaluator and decision-makers 
consciously attempt to avoid this problem, and since both wish to have the pro- 
gram evaluated objectively, the problem will not affect the evaluation’s objec- 
tivity. On the contrary, it is contended that through this method the evaluator 
will be intimately familiar with the program, and therefore, be in a better posi- 
tion to evaluate it objectively. 

Finally, note should be taken that resources were targeted in such a manner 
that each child would be assessed rather chan a random sample, thereby precluding 
a pre-test — post-test design. The reasoning that went into this decision was 
that if the program was to have maximum effect, each student’s individual needs 
must be met. Project personnel, in order to meet these needs, must have relevant 
data on each child, rather than the mean of some theoretical population. 

In closing , let me enter a plea that a general model of evaluation needs to 
be developed if we are to approach maximizing the benefits of the exemplary pro- 
grams. For only by direct comparison of elements that are common to exemplary 
programs can the efficiency and the relative effectiveness of the exemplary pro- 
grams be convincingly conveyed to the public. We hope that the model which has 
been presented here might at least serve as a starting point for the development 
of such a general evaluation model. 
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THE STRUCTURAL COMPONENTS OF THE EVALUATION MODEL 




FIGURE 2 

THE APEX EVALUATION MODEL 
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The use of the word ’’craft" in the present paper is a deliberate one. If 
one checks the dictionary he will find that craft is "an occupation or trade re- 
quiring manual dexterity or artistic skill." It suggests expertness in workman- 
ship, (Let us ignore for the moment the fact that some observers might suggest 
the obiiquitous thought that another definition of craft is ofttimes more appro- 
priate: "skill in deceiving to gain an end; trickery; guile.") To further clar- 
ify the term in the context of the present discussion, one may ask: "Is evalua- 

tion an art?" It is not, because an artist j.s a creator, while a craftsman is 
one who utilizies existing patterns or combinations of existing patterns excel- 
lently. Thus, this paper will deal with workmanship, with craftmanship , and not 
with creativity. 

The current emphasis upon exemplary programs in education is part of a larger 
trend of comprehensive approaches to social problems. A typical program involves 
a number of inter-related activities of which a formal classroom project is only 
one of several that are undertaken. From a basic research point of view, these 
multi-faceted programs offer distinct advantages. Students of social change have 
long maintained that formal education does not take place in a vacuum; instead, 
they have demonstrated that employment patterns, customs, attitudes, value orien- 
tations, patterns of discrimination, and a host of other variables can be visu- 
alized as a system in which change in one area will produce corresponding changes 
elsewhere. Sound as this approach may be, it poses very real problems for the 
evaluator. In fact, the evaluator is never without research problems even when 
the focus is upon a single program in a single classroom. For example, assessing 
the effect of an experimental learning device in a few selected classrooms can be 
a tricky undertaking if an attempt is made to control all of the variables that 
conceivably might be relevant. These problems are compounded when the evaluators 
are called upon to assess the effects of multiple programs of a varied nature, some 
with more measurable products than others. Despite these problems of evaluation, 
such comprehensive projects are not going away. In all likelihood, their popular- 
ity will increase not only in the United States, but also in other nations. Such 
a trend presages the need for evaluation strategies that are relevant to this type 
•of action program. 

In the ensuing discussion, I shall attempt to develop three points: First 

of all, contemporary views of evaluation; second, a rationale for evaluation; and 
third, a discussion of the translation of theory into practice. 
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Contemporary Views of Evaluation: Strength and measures evaluation has been 

defined facetiously as a job, required by authorities on high, that consumes large 
quantities of time, effort and money, but has little effect on actual practice. 
Frankly, that may be the best definition around. Thus, when Egon Guba recently 
scored contemporary evaluation practices in an article entitled The Failure of 
Educational Evaluation , he found even the definition of the term to be inadequate. 

There are three views on evaluation In current use. The first tends to equate 
evaluation with measurement. Historically, evaluation grew up in the shadow of a 
measurement movement. The science of evaluation was viewed as a part of the sci- 
ence of instrument development and interpretation. The disadvantage of this per- 
spective is that it implies a narrow and mechanistic approach. Evaluation tends 
to be limited to those variables for which psychometricists and other researchers 
have developed instruments. The other variables, that is those which can not be 
measured by existing instruments, tend to be viewed as intangibles, a term which 
has the effect of saying that because they can not be measured they have no impor- 
tance* There are parallels in the social sciences. For example, anthropologists 
tend to describe a primitive group as one which, by definition, lacks a written 
language. The Maya, however, are usually classified as a primitive people not 
because they do not have a written language, but because scientists have not been 
able to decipher their writing. So, they are "primitive." 

A second view equates evaluation with determining the amount of congruence be- 
tween objectives and forms. This concept grew out of the work of Tyre and others 
and has had an enormous impact upon evaluation. With this perspective came an em- 
phasis on assessment, not only of product, but also of process (i.e., what actually 
took place in order to produce the outcome). This required the evaluator, or as 
the case might be the administrators, to develop a set of objectives that were 
sufficiently operational to assess the required congruence. Both of these defini- 
tions can validly us criticized on the following point: data become available only 

at the end of a long instructional period. The full possibilities of a particular 
project are thus not only realized, but the form of the definition freezes evalua- 
tion as a terminal event. If process data are available, they can only be utilized 
the next time around. It is too late to use them for the refinement of the on- 
going program. 

The third view equates evaluation with the judgment of a panel of experts or 
judges. This viewpoint has certain advantages. For one, the evaluation can be 
quickly arranged; second, the evaluators are typically experts and can bring a 
great deal of experience to the problem without being constrained by formal instru- 
ments; third, the relationships between several factors can more or less be auto- 
matically accounted for, thus avoiding the delays (often long and time consuming) 
that are associated with aggregating and interpreting data. However, if one has 
a team of experts or judges, these pitfalls are avoided. There need be, there- 
fore, no lengthy time lag between the investigation and the rendering of the judg- 
ment. 

The disadvantages of this definition or viewpoint are just as obvious. In- 
formation about reliability and objectivity are typically lacking. In fact, the 
findings of a panel quite frequently are suspect on the grounds that emotional or 
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personal factors enter into the decision. It is the uncertainty and the ambiguity 
of evaluations which are based upon this research that lead to a feeling of dis- 
quietude about approaching evaluation in this manner. 

Each view of evaluation implies a methodology. It is safe to say that each 
of these perspectives and the methodologies they imply has inherent strengths and 
limitations. The burden of a good evaluator is to be a discriminating eclectic. 

That Is, to look for ways to use the strengths of each perspective for each parti- 
cular research task. 

In the comments that follow, I shall make some specific comments on this 
point. As a preliminary definition, I shall define evaluation as the processes 
that are involved in describing goal-oriented action and assessing its consequences. 
Put another way, it is the interpretation. 




A RATIONALE FOR EVALUATION 

It may seem gratuitous to ask "why evaluate?" I suggest, however, that in- 
; quiring into the functions an evaluation serves can also be very useful in deter- 

\ mining exactly how one shall go about making an evaluation. 

1 In the first place, an evaluation may provide answers to such pertinent ques- 

; tions as the following: 

]. 

j (l) Does the program work? (i.e., Does the program contribute to an attain- 

; ment of the program's objectives?) 

j (2) Is the program efficient? (i.e.. Is the particular program under study 

the best way to obtain results or are alternative methods as productive as this 
particular program?). This question involves the ratio between effort and perfor- 
[ mance divided by input. It also bears upon the issues raised in the congruence 

[ definition of evaluation (i.e., how closely does the action correspond to the ob- 

! jectives desired?). 

' (3) Is the program adequate? In other words, does the program have an im- 

i pact upon total needs? For example, a job placement program might successfully 

j recruit and place fifteen people per year. It might do a marvelous job with the 

; fifteen, and from all over the country people could come in to see this project 

i that successfully recruits and places fifteen people. However, suppose 15,000 

( young people in the same area needed similar help; if the program were measured 

(. in terras of the total needs of the 15,000, it would hardly be judged adequate, 

i although it was effective with the fifteen. 

> (4) What Is It about the program that works? For example, it has been re- 

[ ported that one of the primitive tribes in Brazil concocts a liquid consisting of 

1 various ingredients into which they then dip their arrows before going hunting; 

it is quite effective in stunning the animal hit, thus permitting its revival af- 
ter capture. The tribe puts all kinds of things into this potion — spiders, cob- 
webs, leaves from trees, and numerous other concoctions in addition to various 
herbs. However, not long ago scientists who went to live among these people found 
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that only one little herb was doing the trick and the other fifteen ingredients 
were actually diluting its potencyl 

So, a legitimate question that can be raised in any program Is, "What is It 
about the program that is working? 11 There may be many things one is doing in his 
project, things which he may think are very worthwhile, but the evaluation should 
have answered the question "What Is actually obtaining the objectives?" In some 
cases, success may not depend on anything inherent in the program itself. Instead, 
success may derive from the fact that the program participants believe that the pro- 
gram will succeed and they begin to cooperate with it, and it is that fact alone 
which contributes to the success. 

Second, an evaluation may be used to legitimize a program. The very life of 
a particular project may depend on a favorable climate of public opinion, the good 
will of Congressmen, or a favorably impressed key administrator. The real purpose 
of an evaluation request from an administrative point of view may stem from the 
need to say to some constituency that a responsible research team or a responsible 
evaluator has weighed a project and has not found it wanting. That's all that may 
be desired. An administrator may look upon a rigorous research effort, carefully 
developed instruments, and long schedules of interviews as troublesome and poten- 
tially risky demands upon an already ov'ercrowded agenda. The researcher should 
not dismiss this viewpoint out-of-hand as unworthy simply because it is not scien- 
tifically pure. It Is a fact of political and economic life that needs to be taken 
into account all through the evaluation. An engineer can over-build a bridge, and 
a researcher can sometimes provide information that is neither wanted nor needed. 

If legitimation is all that is wanted, the evaluation as an assessment by ex- 
perts may be an appropriate methodology. If, however, the responsible constituency 
is scientifically sophisticated, if they want pertinent information about the pro- 
gram's process in order to replicate it elsewhere, or if they are likely to demand 
objective proof, this perspective by itself will be inadequate. 

In the third place, evaluation activities may have salutary effects upon the 
program itself. This is implied in one of the more recent definitions of evalua- 
tion (i.e., that evaluation be thought of as a process of acquiring and using in- 
formation for making decisions associated with planning, programming, implementing, 
and recycling program activities). If the evaluator shares information with ad- 
ministrators during the course of the project and soon after the program has been 
completed, he provides them with a basis for judicious decision making. 



TRANSLATION OF THEORY INTO PRACTICE 

I would like to provide a concrete example, from our recent experiences of 
what has been discussed to this point, of the translation of theory into practice. 
As an application, for a three year period beginning in 1967, the Center was heav- 
ily involved in evaluating a Federal effort called "Concerted Services in Training 
and Education (CSTE)," which was a national program designed to develop social, 
educational and economic resources in rural areas. This assignment provided a re- 
search team with valuable experience in translating theory into practice. Evalua- 
ting a project such as "Concerted Services" parallels in several important respects 
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an evaluation of an exemplary education project. In order to demonstrate this, 

I shall briefly describe the "Concerted Services Project." 

"Concerted Services" is essentially a direct-action program which attempts 
to stimulate area development through coordination of appropriate services at 
the local and national levels. At the national level, there is a task force and 
a liaison officer; at the local level, there is a local coordinator with a small 
staff who tries to bring together agency peopJ -s and local leaders through area 
analyses, educational activities, and project development. One of the major re- 
sponsibilities of the coordinators is to bring appropriate people together for 
coordinated action. Another task is to initiate new educational and training 
activities that appear to be indicated from studies of the area. Our task as 
evaluators was to describe and analyze what was occurring; we were to assess its 
impact upon the area, and we were then to report periodically our findings to re- 
sponsible officials in Washington. 

The C:TE projects were essentially comprehensive approaches that involved 
several types of interrelated activities. And I see an exemplary project as 
being just that. It coordinates other activities, and secondly, it is based up- 
on a broad base of experience. 



A PRACTITIONER'S (CHECK LIST) EVALUATION 

Without pressing the analogy further, I want to discuss several points which 
are Important from the point of view of a practitioner of evaluative research. 

First of all, a person should be concerned v/l th the time limits of the evaluation. 
The information should be available when the decision-maker needs it. Like many 
evaluators, I came to my first evaluation assignment strongly influenced by ex- 
perimental design methodology. Hypotheses were to be tested by randomly assigning 
the population into categories, several of which received treatment, while treat- 
ment was withheld from others. Thus I was to have experimental and control units. 
Until the experiment was concluded, I was to resist the temptation to apply any 
measures to the control units that were found to be successful in the experimental 
ones. I was not, in other words, to contaminate the research by releasing any of 
my findings. For this would jeopardize, at least in my opinion, the evaluation. 

I am indebted to the work of Daniel Stuff lebeam , Director of the Ohio State Univer- 
sity Evaluation Center, for correcting this notion. He has correctly pointed out 
that this application of experimental design to evaluation conflicts with the prin- 
ciple that an evaluation should help improve the program. Evaluation is research, 
but it is applied research. If the evaluator waits until the program is terminated 
before releasing his findings, it is often too iate to apply corrective measures. 

We thus adopted the policy of releasing periodic reports to the program adminis- 
trator; and we were continuously cranking out interim reports, advance reports, 
and the like, as well as making trips to Washington to discuss findings. 

Second, in regard to timing, the evaluation should coincide with the Initia- 
tion of the project. Very often, program administrators drift into the policy of 
devoting all of their energy to getting the project off the ground. Then, once 
things have begun to level off a bit, they say to themselves, "Well, you know, I've 
got to think about evaluation." That approach poses some real problems. For one, 
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perhaps the possibility of establishing a base line for a later comparison lias 
been eliminated. All too frequently when a team of evaluators is called in, it 
is felt that if they are sharp enough they should be able to come, assess what 
has taken place, and quickly judge whether or not it is a success. I believe 
that is too high an expectation. 

The logic of proof dictates that an evaluator attempt to demonstrate not 
only what occurs in experimental and control groups, but also what has taken 
place before and after the application of a particular treatment. Now suppose 
one is an evaluator and he finds himself in such a spot; that is, he got started 
on the evaluation after the program was implemented. Sometimes reconstructing 
the base line is not possible. A great body of statistical data about various 
populations is collected by government, business and private agencies. Sometimes 
the memories of informed persons can be relied upon to give a reasonably satis- 
factory picture of what the earlier conditions were. Other sources of informa- 
tion are newspaper accounts, interviews with reporters, editors, public officials, 
government agency reports and specialized investigations (such as dissertations). 
For example , the evaluator may be lucky enough to find out that five years before 
some person working on a Master's degree came in and did a study of the area and 
that study can be utilized to develop a beautiful base line, from which one can 
then show a degree of change . 

In the case of the CSTE evaluation, we actually began after the program had 
been initiated; thus we had to reconstruct a profile of each pilot area from the 
information sources just mentioned. (I might add that toward the end of the pro- 
ject we found out how wonderful a source of information the statisticians asso- 
ciated with the Employment Security Division of each state can be. For they not 
only had accurate information, they even worked some of it out into a program for 
us. We are certainly indebted to these men.) 

Third, one must be concerned with the credibility of the evaluation; that Is, 
the evaluation should be believable. This consideration becomes particularly im- 
portant when the project involves a controversial approach, or if for some other 
reason, it is likely to have outspoken critics. Internal evaluations have their 
distinct advantages, one of which I shall mention briefly. If one evaluates a 
program himself, it is likely that he can apply some of the findings immediately 
to his own program. However, despite this and other advantages, the evaluation 
is likely to be viewed as suspect, especially if it can be shown that the evalua- 
tors have vested interests or commitments in the project. And almost always, this 
is the case. So a person says, "Well, sure they found it to be a success because 
it was their program." In the CSTE evaluation, all the evaluators had advanced 
degrees in the social sciences and were associated with institutions that were 
not directly connected with the project. Hiring persons of known competency and 
impeccable credentials involved paying slightly higher fees, but the product of 
their research proved to be acceptable. 

As it turned out, one of the reports that we developed was critical of a 
particular department of the state government. That report, once it was released, 
was vigorously attacked by the head of that department. Had it been produced by 
a novice, the research might have been rather easily demolished. But as it was, 
the persons who were associated with it had competencies and credentials, and they 
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had no vested interest. They said, "This is what we saw," and told it "like 
it is . " 

Fourth, what is the validity of the evaluation? In other words, the infor- 
mation provided by the evaluation should answer important questions. The re- 
searcher can undertake his evaluation without indulging in needless value judg- 
ments if he couches his study in the terms of the program objectives. These 
objectives, once they have been operationalized, can then be measured in a sci- 
entific manner. It may then be necessary to identify the underlying assumptions 
of certain objectives. Failing to do so can produce an evaluation that could be 
likened to the measurement of the number of times a bird flaps its wings without 
attempting to find out how far the bird has flown. And some evaluations concen- 
trate on the flapping of wings rather than giving attention to direction. For 
example, this would be the case if, in an evaluation of the efficiency with which 
polio vaccine is administered at clinics, no attempts were made to learn whether 
or not the project had reduced the number of polio cases in the country. So, does 
the evaluation really get at what needs to be answered? 

The question of research design pops up again at this point. Researchers 
typically aim at a pre-test, post-test, control group, and an experimental group 
design. This usually has the effect of demonstrating that "X" is greater than 
"non-X." Suchman has correctly pointed out that an evaluation design is greatly 
improved If the evaluation focuses upon specific components of the program rather 
than on the existence of the program itself. This then involves trying to assess 
not whether "X" is greater than "non-X," but rather, whether "X-l" is greater than 
"X-2" is greater than "X-3" is greater than "X-4." So one simply manipulates com- 
ponents in the several programs to find out the source of variants . 

To illustrate this point, the CSTE projects were initiated in the pilot areas 
Without a great deal of fanfare. It was felt that if the persons came in rather 
quietly and began the programs, the agency heads that were already in these areas 
would more likely work with the coordinators. They would not feel threatened; 
they would not feel that someone was coming here to get a lot of publicity. During 
the course of the evaluation, we found that a number of key people in the pilot 
areas did not know of the existence of the project and thus had not utilized the 
program because they did not know it was available to them. We called this to the 
attention of the program administrators, who had to decide whether or not to con- 
tinue the present policy. Because the programs were attaining some of their objec- 
tives , they said "they are successful, they are working". We recommended that when 
they initiated new programs in new areas > this component of the project be varied. 
As the program expanded in some of the new units the project would be given wide 
publicity, whereas in others it would be played down to see if there was any dimi- 
nution in the effects. All I've said here is that this is an assessment of whether 
"X-l" is greater than "X-2" is greater than "X-3." 

The final point Is reliability. The Information provided by the evaluation 
should be based upon indicators which are accurate. An evaluator may be called 
upon to investigate a project where the personnel are attempting to camouflage a 
failure. They are trying to do a good job of covering up and they may be good 
enough at it to mislead even an experienced and astute observer. If the evaluation 
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is intended to provide information for expansion or implementation of similar 
programs elsewhere, the program administrators would do well if they were able 
to structure the project so that its personnel would freely discuss failures as 
well as successes. This can best be accomplished by providing the project staff 
with a measure of job security. 

Say, in effect, "this is an experimental program; we want you to tell about 
the failures as well as the successes. It has nothing to do with whether or not 
you lose your job." Evaluators who call this point to the attention of the ad- 
ministrators early in the project are likely to increase the possibilities of re- 
ceiving accurate information and, conversely, decreasing the possibility of get- 
ting snowed. This proved to be the case in the evaluation of CSTE. The coordi- 
nators had been told from the beginning that their’ s was an experimental effort, 
and they were to share all the information with the evaluator. Our researchers 
had access to the files and all the meetings; they heard about failures and suc- 
cesses. The result was that the evaluator, even though he maintained a distinct 
role as evaluator, came to be viewed as a contributor to the total program effort 
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I have given you a sort of road map, listing the twelve functions of voca- 
tional education. The one function I want to talk about this afternoon is Number 
Twelve - "evaluation." I hope to give you an idea of the way in which we are pro- 
posing to break down this evaluation function into the various sub-functions which 
we believe have to be implemented into an evaluation program. 

As we said yesterday, you have to begin with your program goals and purposes. 
We have talked about specific objectives for two days. They are a must and there 
is no way around them. After your objectives have been stated, you must then for- 
mulate criteria. In other words, what are you going to do to measure whether or 
not these objectives have been obtained? The reason I have a two-headed arrow go- 
ing here is because we believe that in the planning process, there must be a two- 
way flow of information between criteria and the formulation of objectives. The 
reason is very simple. If you write a specific objective and then you get on to 
the task of formulating criteria to measure the effectiveness of this objective, 
and you find that you can f t come up with criteria, there is something wrong. Gen- 
erally, this means there is something wrong with the objective. So, rather than 
fight with criteria until you are out of your mind, back up and take a look at your 
objectives. Rewrite them, if necessary, to the point where some criteria can be 
developed. 

Once you have gone this far, then you have a go/no-go situation. In other 
words, you finally have measurable objectives. Then, you can determine 1) the 
kinds of data you must collect and 2) the kinds of instrumentation needed to col- 
lect these data. Analysis of these data with the results will allow you to compare 
the projected outcome with the measurable or observable outcome to determine the 
difference between the two. If you find you have discrepancies, you then want to 
try to find out the reasons for these discrepancies. If you can determine the rea- 
sons, you then have material you need to make decisions; you can then re-examine 
and reformulate program goals, as well as specific objectives. If you find that 
you don't have the objectives you need because you can't formulate criteria, you 
then have to examine program goals and objectives, because these objectives may 
have faulty goal statements. It may also be that you need additional baseline data 
to develop specific objectives or overall program goals. 



1 



\ 



I would like to make a plea, at this point, for all of us in vocational edu- 
cation to place particular emphasis on product evaluation. If there has ever been 
a shortcoming in evaluation, it is that we have been overly concerned with process. 
We are wrapped up in process to the point where many of us have neglected the prod- 
uct. We may spend all the time we want on process, but if it doesn't turn out that 
product, we are still in trouble. I am not saying that you should ignore process 
evaluation because in exemplary programs you are testing new processes, and trying 
to develop new ways of attacking old problems and hopefully coming up with work- 
able, reasonable solutions to these problems. 

In looking at this business of product, we have to accomplish two tasks. One 
of these tasks is to identify the learning which must take place before you can 
meet your purposes, your goals, your objectives. Traditionally, in all areas of 
education, we can be criticized for our work in the area of curricula development 
for one simple reason. A lot of our curricula development has been completed for 
the wrong reasons — for example, a gap in the school schedule. We have gone this 
route, or similar routes, too often. We have ignored this business of analyzing 
the learning task which the learner has to accomplish in order to reach the pur- 
poses or the desired output or the end product. We have to also take a good hard 
look at input competencies. What is the level of competency of the learner when 
he enters this program? We must take this look before we will have a fighting 
chance of projecting our expected outcome. We also will have a better chance if 
we can identify input competencies of the learner and the ways and means to attack 
his particular problem. If necessary, we must completely change the system so it 
will fit this individual learner. It gets back to the fact that people involved 
in education can no longer afford to be simply imparters of information. It is 
too expensive. We have to become learning managers. 

In looking further at evaluation, there are a number of areas which have been 
identified and which must be looked at in depth. The first is what is called pre- 
program evaluation. Pre -program evaluation is a function primarily of a planning 
unit, that is, the people who are doing the planning. Here you must get back to 
this business of writing the program objectives, writing the criteria, and checking 
them back and forth until you are sure you've got everything wired to produce ex- 
actly what you want. In other words, evaluation takes place in the planning phase 
of the operation. Everyone tells us that evaluation is a continuous on-going func- 
tion, and well it has been. However, the one area in which we traditionally fall 
down .is this area of pre-program evaluation. Evaluating the planning process, the 
function evaluation, the on-going evaluation, or whatever you want to call it, has 
been talked about in great detail. This can be either a function of the project 
manager or of the contractor. Terminal evaluation is a type that has to be done on 
an independent basis. Post-evaluation is another area of evaluation which we have 
ignored. We evaluate the end of the program, everything looks good and so we move 
on — blissful in our ignorance — and all of a sudden someone comes back and asks about 
the people you are turning out in your program. Perhaps they qualify for jobs but 
once hired, can't hold them. Therefore, I think we have to take a look at seme of 
our products and processes at some period in time after we have finished with them 
in our programs, to see if the bit of knowledge or skill which we have imparted is 
of a type that will stay with and benefit our students. 
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Exemplary programs are supposed to have independent evaluations. This means 
we must go outside of the organization for our evaluation. As potential project 
managers, one of the things you should be concerned with is this business of pre- 
paring RFFs, because, as I interpret it, the responsibility for developing a spe- 
cific evaluation model is given to the contractor. What should we be looking for 
when we write a RTF? We've had some experience with them and this is what we have 
found. First of all, know exactly what you want and be able to spell it out spe- 
cifically in terms of how you want it accomplished. The accessibility of data is 
very important to a person who is writing a proposal for you. If you have told 
him what you want done, and how large your project is in terms of schools, students, 
etc., but you have not told him about the accessibility of data, it will be impos- 
sible for him to budget this thing out, because he will have no idea of how much 
staff time he will need to expend in collecting the necessary data. You should be 
concerned about a time line. How much time are you going to grant your contractor 
before you ask him to come up with a final report? The specification of a final 
report is something that is many times neglected in writing a RFP . We all assume 
that our contractors will prepare a final report for us, but it should be specified 
in the contract. Also, how are you going to pay him? This is equally important to 
a contractor. Are you going to pay him 20% at the end of the first month, and 60% 
upon the completion of collection of data, and the remaining 20% upon acceptance of 
a final report? He must know in the contract how you are going to pay him. 

In looking at your requests for proposals and deciding where they are to go, 
you have a problem. I am sure that all of you are either connected with univer- 
sities or have connections with universities . A university is one possibility. The 
other possibility is a private organization. There are advantages and disadvantages 
to both which I think we should consider. Generally, when working with a university 
you will contract with people you know. You will know what to expect and you know 
how they operate, so you have a pretty good idea what the final product will be. 
Generally the university will have a working knowledge of vocational education and 
will be familiar with the project on which you are working. On the other hand, 
there are disadvantages in working with the university which you will not have in 
working with private companies. One disadvantage is that when you contract with a 
given department, you contract the services of the total staff. We all know that 
every university department has outstanding staff, the same as they have mediocre 
staff, as well as even poor staff. With the university, you are limited to the staff 
that is available to you for evaluation. The private contractor will usually have 

a small staff and they will contract with people who are usually outstanding in 

their fields. I say usually, since this is not always true. When a private con- 
tractor gets a contract similar to the one you may be asking for, he can tap people 
who are on retainer to his company to come in and assist with the job. With the 
private contractor, you have his full commitment to the project. This is his full 
responsibility. If you are contracting with a university or another educational 
agency, you have to remember that all members of that staff have other responsibil- 
ities. Therefore, you may have some time lag. These are some things we have dis- 
covered in dealing with both universities and private organizations. 




The thing you should determine in your own mind, if you go the private con- 
tractor route, is whether or not the potential contractor is financially respon- 
sible* There are a great many reputable companies, but, by the same token, there 
are a lot of "fly-by-nights.” 

Some people have the tendency to use the term "in-house" evaluation for in- 
ternal evaluation; and after an experience with an external evaluation, they have 
a tendency to call that an "out-house" evaluation. The reasons for this are many. 
The advantage of an "out-house" evaluation is the fact that your contractor has 
no emotional involvement. You are with the program; you are married to it; you 
can see no faults, literally. Yet, as the result of lesser emotional involvement, 
"out-house" evaluators are more likely to see your program as the clients or the 
students will see it. These evaluators use only the written, stated objectives 
which you have furnished them* If we evaluate our program ourselves, our evalua- 
tion tends to become a big white-was.i job. Also, an "out-house" evaluator's only 
responsibility is the task outlined in the contract. But if you evaluate a project 
yourself, you know what your other responsibilities are and these other responsi- 
bilities may interfere with the proper completion of the evaluation task. 

Evaluation problems which occur are generally caused by a number of reasons. 

One reason may be that you lack baseline data; another reason may be the lack of 
specific measurable objectives and the fact that you expect more than is reasonable 
in terms of the money you are spending or in terms of your objectives. 

You may be asking how much you should spend on evaluation. This is generally 
where we get into problems, We want everything about our program in the evaluation 
and we budget about $500 to get it. This is totally unreasonable. With our educa- 
tion development act program, we have specified that every such project must budget 
and expend a minimum of 3% of its total direct costs for evaluation. A good figure 
to shoot for is 5% of your project costs. 

To summarize, the things about which you must be most concerned, in your eval- 
uation, are measurable objectives. Your objectives have to be legitimate. They 
have to be legitimate in terms of the society within which the educational system 
is working. They have to be acceptable to our clientele, to the board of education, 
to the board of supervisors, and to the governor. They have to be legitimate as 
well as measurable and specific. We have to have relevant criteria--criteria that 
are relevant to the objectives, criteria that will measure the things our objectives 
say we are going to produce. We must have adequate resources. If we do an "in- 
house" evaluation, we must have adequate personnel who know what they are doing and 
who have adequate funds and adequate time for the evaluation tasks. 

If we go "out-house " we also must have adequate funds to get the necessary 
job done. Last, you must have an honest desire to have a really good evaluation, 
and, subsequently, a commitment to use the output from that evaluation. If you get 
something from the evaluation that doesn't look good, don't just sit there and hack 
about the lousy data the evaluators collected. Ninety-nine times out of one hun- 
dred, it will be your own fault that they had poor data with which to work. So, 
look at the recommendations of the evaluation and see what you can do about imple- 
menting those recommendations. 
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Evaluation SumARY 



Dr. Larry Hutchins 
Far West Regional Lab 



Let me explain today what the Far West Laboratory is all about. We are a 
development agency. We sort of think development is our thing and we specialize 
in it. With regard to evaluation, we break it down into three phases. There is 
a preliminary phase, a performance phase, and an operational phase. During the 
first year, we concentrate on developing prototypes. We just don’t believe that 
wc can develop an entire evaluation right at the beginning. In fact, I defy you 
to write your specifications for your third year of operation before you even get 
the program started. You can't dc it. I also doubt if you can do it even after 
the program has been in operation for nine months. So what we simply try to do 
during the first year is to define what we call the preliminary phase of develop- 
ment in which we initially outline our overall goals and specify the components 
of the system and then develop prototypes for each of these components. 

For example, if you are going to have a television class sequence in the 
program, you don’t try to develop all thirty television programs at once. Instead 
you develop two, three, four or five of them the first year to see if the compo- 
nent you were going to use is going Co work. You can write the specifications 
for that much of the program more easily. Then, the second year, we go into com- 
plete specifications for each of the components, using the kind of controls over 
the program which are necessary. In other words, when we are in this phase, we 
are actually there with a very heavy hand in the classroom, making sure that the 
conditions we had outlined are being followed. The third year, we go into what 
we call an operational phase, in which our control over the situation diminishes 
significantly. In vact, we get out of the system to see if the teachers can op- 
erate it by themselves. 

We really ask three different sets of questions. First of all, we ask if 
our goals are worthwhile and are the tasks we have envisioned for the long-range 
plan likely to work. We ask those questions in the first year to determine if 
they work for the prototypes we have evolved. Second, now that we have developed 
them with our control, we must determine if they work under the specifications we 
want. Then the third year, we step back to see if the things we have set up will 
work without our control. I don't know if you can do that with your exemplary 
programs, but certainly if you have to come with a complete specification in the 
first year for a three year system, I wish you all the luck in the world. 
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If possible, you should negotiate each year. Also, it might be a good idea 
to develop an interim kind of report back from the evaluator so you are not faced 
with a really gruesome report at the end of the year. 

I also wish to urge you to please specify the goals of the individual compo- 
nents of your program. You will find that if you can write fairly good specifi- 
cations for these components and if you can track them periodically, then you can 
begin to draw a curve and from that curve determine whether or not knowledge is 
increasing. The fact that you can demonstrate change over a long period of time 
is a very significant kind of evaluation. Furthermore, this way you can also 
identify the components that aren't working for you. At the laboratory, we have 
had really phenomenal luck in being able to determine what will work and what will 



To summarize, if you can break down your evaluation into several phases and 
if you can get the contract officer to agree with this approach, then try to ac- 
complish different tasks in different years. Second, try to specify the different 
components and try to specify what you want them to accomplish rather than try to 
simply design the overall evaluation system. That's fine if you can do it, but in 
the early developmental stages, when you are trying to make something work, you 
really need to be able to look at the various factors you are developing to make 
sure that each one of them is first tracking right. 
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Presentation of a Dissemination fta_ 

FOR 

Exemplary Projects 

Mr. Tom Clemens 
Acting Director 

& Division of Practice Improvement 

National Center for Educational Communication 



This morning, I think it is only fair to let you know to what you are going 
to be exposed. In order to do this, let me first of all tell about the assump- 
tions upon which I will be working. Let me point out that I am not sure I have 
any kind of a model to provide you, for a model implies a kind of rationality 
that may not really exist in the information field. Nevertheless, I hope I do 
have some suggestions and comments that will be helpful, but even more than this, 

I hope what I say will stimulate some thought and result in some suggestions from 
you---because in the final analysis, what happens in the dissemination of informa- 
tion about the exemplary programs depends a great deal more upon you than it does 
upon me or anyone else within the City of Washington. 

My first assumption is that what the exemplary program is really all about is 
not just doing things differently, but doing things better than before in education. 
Unless the pay-off from this program results in an improved learning experience for 
the people of this country, the program is a failure. The program will also be a 
failure unless the best experience of these exemplary programs spreads to be applied , 
adapted, and installed in other centers, so that they have a nation-wide impact. The 
programs must eventually improve vocational education opportunities. 

A second assumption I am making is that the best hope we have of systematic 
and rational improvement of education is through continuing utilization of the 
knowledge base which exists about education , a knowledge base which is continually 
expanding. 1 believe this knowledge base is important to use because one of the 
.'ays educational change varies from those in other fields is that such changes 
•.end to be adaptations of knowledge and experience in other settings, rather than 
just the kind of straight-forward and wooden-head adoption which has occurred in 
many other fields. It’s quite simple for a farmer to adopt hybrid seed corn, but 
where there are the varying constraints we find in the 17,000 local school dis- 
tricts and fifty autonomous state education agencies in this country, we s5moly 
cinnot find adoptions that are as straight forward as those in many other fields. 

So knowledge must be used to stimulate appropriate, rational adaptations of inno- 
vative programs to the unique requirements of the educational settings around the 
country . 
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A third assumption X am making is that you people represented here (particu- 
larly those of you involved in the exemplary programs) are what Evert Rogers would 
call the "cosmopolites," the innovators, and to some extent the opinion-makers in 
vocational education practice. And until these innovative and opinion-leading 
members of a social system adopt new programs and demonstrate their effectiveness, 
these new programs have no chance of spreading through the social system. In this 
case we refer to the social system of vocational education. Therefore, I am looking 
to you to be rational users of knowledge for educational improvement, as well as 
producers of knowledge which can be used by others in improving their educational 
programs . 

I would now like to share with you what we know about how educators use know- 
ledge , what the characteristics of educators are as users of information, and what 
kinds of information requirements educators have. Then we will try to draw up some 
implications about what this means for dissemination programs for exemplary projects. 
I would then like to talk about you as knowledge users, about some of the things you 
might do or might want to consider in your programs. Then, since you are also the 
ones who will be playing the major role in making sure that these innovative ap- 
proaches spread throughout the educational community, we had better consider also 
your role as knowledge producers . 

Very frequently, it looks as if the only thing that educators use information 
for is to justify the decisions they have already made, or else to have a good 
bibliography attached to a proposal for funding by a foundation or the "Feds," 
However, in a knowledge base field, as opposed to an experience base field, infor- 
mation can be used for a number of purposes throughout the entire process of get- 
ting an idea right on through to installing and maintaining it. 

The first thing for which knowledge is required is the diagnosis of what your 
problems are, and then to operationally define them. When you feel your program 
is not doing as well as it can, you must be able to determine how it is or is not 
meeting the needs of your clients. In this diagnostic process, you clearly have 
xo rely upon collection of information within your school system or your organiza- 
tion. It is also very useful to utilize information from outside sources. Find 
out how you are doing in comparison to others, whether other programs relate to 
yours in some way, and find out what kinds of information gathering techniques and 
information gathering instruments are easily available for your use. In rational 
program operation, you need information simply for diagnosing where you are. It is 
evident that, in using information, it must assist us in developing an awareness of 
alternatives to current practice. This, of course, is almost exclusively an out- 
side information source. Most of you are aware of the new possibilities which will 
come from the activities of other programs. However, in choosing among these alter- 
natives, you will probably engage in a little "in-the-head" evaluation to decide 
whether some of the alternatives seem to fit you better than do others. This "pre- 
trial" evaluation requires information about those alternatives which have been 
identified. It also requires that you give some thought to what the unique charac- 
teristics are within your own educational setting. Having engaged in this pre-trial 
investigation to select an alternative, you will then engage in a trial of the inno- 
vation. Most rational adoptions of new programs are based upon a limited trial 
adaption prior to the time that it is installed on a System-wide basis . In the trial 
process, again, you will rely very heavily on internal information sources. Unless 
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you relate this internal information to information from external sources, you 
are not going to be very successful in carrying out an effective trial which 
will lead to an effective education program. Clearly then, after the trial, you 
have to go through another evaluation, a post-trial evaluation. Here again, you 
are going to rely heavily on internal information which you collect. 

Even if, at this point, you can say your program is highly successful, you 
still need two more kinds of information. This is also a place where you will 
have to rely on external information sources. First, you will undoubtedly need 
information about how you can install this on a system-wide basis. There are a 
number of questions that may need to be asked at this time. Where can you go to 
get continuing consultive help to engage in this installation process which is 
likely to take a year to three years? Where are there training opportunities for 
teachers who may have to be reschooled? Where are there guidelines for training 
which may be handled within the organization itself? What are the sources of 
material and equipment? Unless you want this new program to be consistently out 
of tune as your situation changes, you had better find ways to insure that it con- 
tinues to adapt to changing conditions within the school system. These are the 
kinds of information which you as a knowledge user will require in the operation 
of your program. It is hoped that you will provide information designed to serve 
the purposes of other vocational educators. This information should help them 
diagnose, help them become aware of alternatives, help them engage in both pre- 
and post- trial evaluation, help them conduct effective trials, and help them main- 
tain their new program. 

It is all very well and good to talk about the wide range of uses of know- 
ledge, but if this information is to be provided in an effective form that does 
get used, it means that we have to relate to how educators normally use informa- 
tion. When we go into our information program, we must keep in mind the charac- 
teristics of the user. It is impossible for us to set up a system which expects 
the user to change. The system must be organized in such a way that the user will 
naturally make use of the information available to him- 

Some things come through very clear about the educational practitioner. He 
will, first, make use of the most acceptable information source. There is evi- 
dence in other fields, such as engineering, history and the military, that the 
practitioner will make use of the most acceptable information source first, and 
he will also make use of it more frequently, especially if he thinks the informa- 
tion it provides is very good. He will always go to the most acceptable source. 
There is a kind of Gresham’s Law in the information field; bad accessible informa- 
tion will drive away good inaccessible information. So it is therefore essential 
for us to make sure that validated, solid knowledge in education is readily avail- 
able and near at hand to the educational practitioner. 

Second, educators want timely information. They don't want something that 
tells them about the dear, dead days beyond recall. Instead, they want informa- 
tion which informs them about as current a status as possible. Further they want 
it within real time constraints. You may find that for some purposes a two-week 
turn around is essential if information is to be useable. 
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Our third point is relevancy . There are many educational reports which do 
not appear to have very much relevancy to many educational practitioners. You 
may have seen some of these reports yourself. 

A fourth preference of the educator is that he be provided comprehensive 
and distilled information. It is unreasonable to expect a busy superintendent, 
administrator or teacher to be able to read a quantity of technical literature. 

They simply do not have the time and all too frequently individual technical re- 
ports do not appear to relate to what their real concerns are. Therefore, it is 
extremely important that instead of just having individual reports on individual 
projects, we also find ways of distilling, interpreting, and packaging information 
according to the requirements cf educators in the field. Another thing that stands 
out very clearly is that educators rely very heavily on interpersonal communica- 
tion. They are much more likely to make an adoption decision on the basis of talk- 
ing with someone who has been that way in the past, or to someone whom they per- 
ceive is credible and expert, than they aie by reading technical literature. It 
is therefore extremely important for any effective information system in education 
to provide for these interpersonal linkages. At certain points in The adoption 
process, people believe people a lot more than they believe data. Further, people 
can ask questions. Educators can talk back if they are interacting with you. They 
can't talk back to a report. 

Finally, educators put a great deal of confidence in first-hand information. 
There is a great deal of evidence that educators, and administrators particularly, 
tend to make adoptions of those programs which they have seen in operation rather 
than programs which have just been described to them. Therefore, there are certain 
characteristics which have to be built into the information system. We have to 
rely not just on printed media or audio-visual media, but also on inter-personal 
communication as well as on opportunities for potential adopters to see effective 
programs in operation. 

What should be our knowledge base in education? Clearly we have to have R 
and D reports. We do need a knowledge base of E and D reports. This is the most 
mature educational infoi-mation sub-system at the present time. Through ERIC (Edu- 
cational Resources Information Center), which currently has a file of over 35,000 
separate reports on research and related information in education, a monthly docu- 
ment is published. This document provides access to these reports. Through the 
Journal of Education , ERIC provides access to the Journal 's literature on a monthly 
basis. These reports, as you know, are available on microfiche and on hard copy 
from the ERIC document reproduction system. 

We also need reports of your exemplary programs and their products. We do 
not have good or easy access to literature on exemplary programs. We need re- 
ports of what kinds of curriculum guides are available for this purpose. This is 
an area where we have got some work to do. Presently, we are at a "double plus" 
stage on R and D reporting; we are probably about at a "zero" stage on having re- 
ports of validated programs and validated instruction. This is an area in which 
you have a great deal to contribute. As you feed in reports of your exemplary 
programs, you immediately broaden this knowledge base. Hopefully, as time goes 
on, elementary and secondary education and higher education will catch up with 
you in making these kinds of reports available to the educational community. 
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We also need interpretive reports of exemplary programs. Remember that 
educators want distilled information. The communications pregram of the Far 
West Lab is doing a very impressive job with some of its integrated information. 

It is helping educators to know about the characteristics of different kinds of 
educational problems. In the Office of Education, the ERIC Clearinghouse pro- 
duces a variety of information analyses. For example, we have PER]’ kits end if 
you write to your state agency you will find our which ones are uvailabJe. There 
is, however, a need for a great deal more interpretation and distillation of the 
results of both research and practice in forms which are oriented toward decision 
making and designed for the use of the decision implementer of the school. This 
is also an area where we must do a great deal more. One of the resources in voca- 
tional education for developing such interpretive materials is found in the posi- 
tion of research coordinator. Some coordinators have done some effective jobs 
in this kind of interpretation. 

Unfortunately, you do not have either intellectual or physical access to the 
products coming out of research and development activities in exemplary programs 
which have been supported by the Office of Education. We hope to do something 
about this problem. We are currently developing a "publishers alert system" in 
which each project will report those kinds of new products which are developing 
very early in the game. We can notify publishers and they then will have an op- 
portunity to relate to the project director and attempt to negotiate arrangements 
to either make it available or help produce it. This program will be operational 
in this current fiscal year. It should markedly increase the access to products 
coming out of OE programs. 

Also, there is a need for technical assistance for two-way communication. 

The kinds of technical assistance which are now available to you are your R and 
D unit reports , as well as the reports from other sources sent to the various 
directors of vocational education. These perhaps represent reports from some of 
the other specialists in the state agencies . You can also tie into reports from 
the universities in your community. However, we have no systematic method of pro- 
viding technical assistance to educators on a demand basis. This is clearly a 
place where there is need for a great deal more effort, both at the local and the 
Federal levels. I would submit that this might be one of the prime areas of need 
to which you may want to address yourselves in the future. We must be in a posi- 
tion to assure the availability of adequate technical assistance in the process 
of trying out, adapting, installing, and maintaining effective vocational educa- 
tion programs . 

We must find ways of assuring that on-site visits can be pz’ovided for less 
innovative educators so that they can actually see a program in operation without 
jeopardizing the quality of the program. Such a person must have an opportunity 
to interact with people who have been through an exemplary program. If vw can 
provide this kind of product information service, I think we will find there 
will be a much more rapid rate of adoption of tested programs in vocational edu- 
cation. If we do this, I think you will find a marked increase in the rate of 
quality vocational education throughout the country. 



In the process of operating your exemplary program, what might you do to 
make sure your program is indeed exemplary and efficacious? First of all* there 
ore a variety of information sources you can use, Vou should undoubtedly be ty- 
ing in with the ERIC System. One way to accomplish this is through your research 
coordinator or your state educational agency, where they are set up to conduct 
searches for you. In some cases they may even provide you with copies of the re- 
search reports. Vou may even want to have a limited collection of ERIC material 
on hand. ERIC is an important base for you to have but it isn't sufficient in 
and of itself. You should be signed in with such other information mechanisms as 
the School Research Information Service and you should make use of the retrieval 
system for dissertation abstracts. You should also make use of other information 
sources resulting from vour professional associations. If you feel this is impor- 
tant, we in the Office of Education will be happy to provide training sessions for 
you or your staff on how to use information resources. We will be happy to help 
you engage in the methods of search and retrieval. If this is of interest to you 
here, talk it over among yourselves. If there could be a one, two, or three day 
session for all the exemplary program managers , we would be happy to see what we 
could do about setting up a training session to help you in making more complete 
use of available information resources. 

It is important to talk to you about your role as knowledge producers . In 
the work you are doing now, you will be producing knowledge for others. Might I 
suggest two cautions: Don't engage in dissemination too early. Make sure that 

you have something to say before you start saying it. You should write a con- 
tinuing description as to vrhere you are , describing the kinds of activities you 
are conducting. You should let us know what kinds of materials and programs 
you have developed which might be exploitable commercially. However, don't simply 
begin advocating what you are doing until you know it works. If it is not working 
for you, it is not going to work for anyone else. The progress of educational im- 
provement has been set back more by the dissemination of programs where all they 
say is, "I tried teaching and found God.” It is therefore essential for us to at- 
tempt to provide validated information in our dissemination efforts. One of the 
ways to do this is to look at what you are going to report from the viewpoint of 
the person who is going to make use of the information. You have already been 
provided with one repnrt which may be of some value in communicating the results 
of your work to others. The Far West Lab has done an excellent job of finding 
ways to lay out the format for reports of programs in such a way that they are of 
use to the person v/ho has to make program decisions. 

Still another kind of report which may be useful is this one published by 
the Office of Education called Preparing Evaluation Reports — A Guide for Authors. 

It tells you how to develop a report which is oriented toward the user. It de- 
scribes how to write a decision oriented report. Such a report should help the user 
make decisions. In order for a report to assist in the decision-making process, 
it must describe the context of your project. Remember, the reader is going to 
have a make a decision and he is going to want to know that you are enough like 
him in order for him to think your program might have sane relevancy. He will 
want to know about the locale of the program. More specifically, he will want to 
know the population trends, and the economic trends of the school system in which 
the program took place; he will want to know about the organizational structure 
of the program and he will want the project 'ft financial picture. He will also want 
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to know about needs assessment as it relates to the project and he will want to 
know the project's historical background. If he can have this information, he 
will then be able to see your project in context and will then be able to see 
whether or not it relates to his own environment. 

If it does relate, he will then want to know more about the program. He 
will want to know about the scope of the program. He will want to know about 
the personnel who are involved in the project. He will want to know what kinds 
of teachers, administrators, and specialists were required and whether or not 
special training was necessary to get them tooled up for the program. He will 
want to know about the procedure of the program or project. (Each of these 
phases of program description are discussed in great detail in the U.S.O.E. guide.) 

One of the biggest impediments to change in school and educational systems 
in the United States is that they do not have any risk capital. Therefore, the 
question of the project’s budget is going to be extremely important for the edu- 
cator. 

In describing your program, how you present your evaluation will also be 
very important. Your decision oriented user will be concerned with your objec- 
tives, what you were trying to achieve, who the participants in the program were 
and what kinds of measurable changes occurred in this process. That implies you 
may want to give some thought to what you tell them about how you analyzed your 
data. You will also have some concerns about how you present the data in a form 
for people who are probably not evaluation experts. Unfortunately, most of the 
reports on innovative activities are so concerned with the findings that they 
neglect all other aspects of program description. They rarely tell us about the 
major changes in the program, and they rarely give us much information on the data 
which indicates the reasons for these changes. 

The user will also be interested in your recommendations. This is another 
area of the report which should be oriented toward the user. 

Accept the fact that people are going to want to come and see your program 
in operation. I would strongly suggest you try' to defer any kind of site visita- 
tion until such time as you know what your program £s worth. One of the things 
that will foul up an exemplary program more than anything else is having large 
numbers of visitors climbing all over the students and peering over the teacher's 
shoulder. I would suggest it is advisable to try to hold off visitors for from 
six months to a year. If you do not feel your program is going well at that point, 
don’t allow them to clutter up your efforts until you are in a position where you 
can handle them. When you feel your program is operational and you can handle 
visitors, you probably will need somewhere between one-half and one and one-half 
resident guides in order to take care of these people. You will want to develop 
some descriptive materials which they can take away with them. You may want some 
kind of audio-visual documentation of the program. This might include visual or 
audio recordings of what actually happened in classroom situations. In some very 
elaborate ca:=er,, there are institutions wnich have set up one-way mirror observa- 
tion rooms oc closed -circuit TV systems in order to seat visitors in a room or 
rooms separate from the youngsters in the actual classroom. 
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Your concerns for your visitors should be threefold: 1. Provide them 

with accurate information; 2. Respond to their questions rather than telling 
them only what you want them to know; 3. Keep the integrity of your program by 
not allowing them to interrupt the program. Some fraction of those peoDle re- 
sponsible for visitors should be topfi Ight members of your team. For example, 
a professional should be able to interact with your visitors, answer their ques- 
tions, and get two-way communication. 

You can get yourself eaten alive and your budget can be completely consumed 
by sending reports to everybody and his brother who writes in and asks for infor- 
mation on your program. When you have agreed as a group on what the nature of 
your report should be, I would then suggest you have two reporting channels: 1) 

Directly into the U.S.O.E. Bureau of Vocational and Technical Education, and 2) 
Directly into your state vocational education agency, iou can then handle the 
requests for information by simply referring them to your state agency or by re- 
ferring them to the ERIC system. 

At the present time, there, are only about thirteen state agencies that have 
any kind of comprehensive knowledge base* However, there are many efforts of 
this kind being carried on by the research coordinating units. I would suggest 
that we work to get comprehensive information service in all our state education 
agencies. Only then will thw research coordinating units be able to do those 
kinds of things you need. They can then begin interpreting, distilling, and 
tailoring the knowledge base to th*s ?equirements of the local educator. Unless 
we take this approach, we are going to have such an unreasonable demand for in- 
formation in education that we will never be able to meet it. There is a seller's 
market right now and it is going to get worse. I would therefore strongly urge 
\ our doing everything we can to build a comprehensive information base and re- 

1 trievable system in the state agencies which can then be used and exported by 

specialists and experts like our research coordinating units' staffs and other 
professionals in the state agencies. Only if there is a feedback through the 
system in such a way that we make sure our information services are relating 
back to the requirements of the user at all points, will we be able to get edu- 
cation to K',.ure it is knowledge-oriented rather than experience-oriented. I am 
proposing that you can play a part in our goal to provide education with access 
; to all relevant reports in the world. We hope to do this in a few years and 

} it is closer than you think. 

• It is even more important to have on hand an array of refined information 

products. We want a system where each practitioner does not have to do the job 
all by himself. We also want a system where the user will have help in applying 
’ knowledge through technical assistance consultation. However, in order for this 

■ to happen, we need to provide useable information when needed and where needed. 

i I have proposed some ways in which the information emerging from your pro- 

grams can be useable. We will need a national network of information systems to 
accomplish the task. We will work on it; we will also need your advice and assis- 
tance. It will have to be operated by skilled people and we don't have enough of 
I them. We are, therefore, going to have to make sure that we get double duty from 

j those people who are available for the job. If they are provided the proper re- 

; sources, it will be possible for us to have a changing and more responsive educa- 

; tion system— a system which is moving toward the attainment of specified goals 

; much more effectively than it has moved in the past. 
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You have had models thrown ar you for a few days now and they are probably 
running out of your head. I guess every speaker has his own model, and I’m not 
going to oe any exception. So, let me start by putting a model on the board 
simply with a few letters. "RODPC” means "Research, Development, Diffusion, 

Practice, and Consumption" . I think it is meaningful to begin to break it down 
into these components. Let me ask you, "Where do you fit into this cycle?" Re- 
searchers — X don't think that by any definition you see yourselves as researchers. 
There is. an agency within the Office cf Education which, I understand, will be 
supplying money co people in this area and perhaps you in another capacity will 
fulfill that role. Nevertheless the project you are operating is not really a 
research project. A research project is designed simply to create new knowledge 
and absolute truths. Somewhere, beyond this point, you are going to act on a lot 
of ideas you hope are truths* I don’t think you are going to have the opportunity, 
money, or resources to demonstrate which kind of a truth with which you will be 
working . 

Your responsibilities begin somewhere after that. Just to make it a little 
more dramatic, let me cross the "R" off. Let me go to the other end now and discuss 
"Consumption." The consumer in most educational environments jr. yoyr student. And 
I'm going to cross that off, too. You know it’s very funny that they are the last 
people to get involved in a project. They are the ones who are presumably going to 
benefit, but they are never involved in the planning*, they are never asked what goals 
they have. This is a sad state of affairs, and in some places it is changing. We 
do a lot of reviewing of educational development and as I cross the "C" off, I would 
like to comment on one phase which I think can have a potential for your project if 
you are willing to entertain the idea. Individualized Instruction is one of those 
catchwords today, but it is certainly something of which you all have heard. It's 
a very complex thing though*, think about it for a minute. Individualized instruc- 
tion has at least two kinds of components into which at least three or four elements 
may be included and you can put the thing together as many ways as possible. You 
can define goals for students, you can define the method by which they are to learn 
and you can define the rate or the timing by which they are to undergo this learning. 
Who defines these things? Traditionally, it has been the teacher and the school sys- 
tem. In individualized instruction, this changes*, the student increasingly has a 
role. There are programs today which do allow the students to select goals. There 
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are programs which allow a student to select his own methods. The instructor 
gives him an objective which he prefers. Then he must go out and do something, 
that is, !, learn by doing." There are students who learn by doing through reading 
and prefer that as a mode of instruction. They don’t all have to be out there 
working with their hands. Just exactly what involvement are you going to allow 
the consumer in your system? I wanted to talk to as many of you as I could to ask 
about what you are doing in your projects and of the few people I did talk to, not 
one of you mentioned that the consumer was going to be significantly involved in 
selecting his own goals, his own methods. I hope that was just an oversight, and 
I hope that it is not true. But it is unfortunate that in most projects, in most 
traditional school settings, we cross the "C" off in the planning process. 

Well, that leaves "Development, Diffusion, and Practice." Certainly you are 
in the practice end of the continuum. You all will be working with school children 
in school settings, so that I can’t completely cross the M P" off. But as I par- 
tially cross it off, I would like to make one comment. As I understand it, a num- 
ber of you are going to try to work first of all with the regular curriculum. You 
are going to be in for a rude shock if you have never worked with elementary and 

secondary teachers. You are going to have your hands full. It’s not because they 

are stubborn, not because they are not knowledgeable, it's just that they need help 
The average elementary school teacher is overwhelmed ; she has thirty little kids, 
and it's all she can do to manage the day and come out sane at the end of it. And 
she wants to get home at 4:00. She may have the best motives in the world to help, 
but she doesn't have the time. So unless you can put into her hands pre-planned, 
pre-packaged material, you are not going to have very much success with her. She 
will do whatever you want her to do if you give it to her in such a way that she 

can work it into her schedule without a lot of upset. Again I do want to say, they 

do want to help, but they have to be given things in a specific way in order to do 
it. 



So now, what I’m really getting down to is that you are really responsible 
for both development and diffusion. When I first started talking with you, I did 
not hear the kind of things I was hearing this morning. "What's our obligation 
to diffusion?" I think most of you saw this as an opportunity to get some money, 
bring it into the local school district, make some improvement in what you were 
doing, and you hadn’t spent a lot of time considering whether any other school sys- 
tem was going to benefit from it. If you are really going to be responsible for 
diffusion, you've got a special obligation to this task, one that is not going to 
come easily and one about which we don't know a great deal. Nevertheless, we do 
have an obligation, and I believe that if your programs are to be successful, you 
are going to have to live up to this responsibility. 

I’m now down to the body of my speech which addresses itself to "Development 
and Diffusion." 

Let me talk first about Development. There are at least three kinds of de- 
velopment which are meaningful to talk about. The first is the kind with which you 
are all familiar; it's kind of a commercial textbook development. They have been 
developing produce -3 for use in the schools long enough that they know h^w to do the 
job pretty well. You may not like what they develop and you may want to change it. 




110 



115 



but they have the know-how. I assume that most oi you are not going to go that 
route, so I will dismiss tax' books. There are two other kinds of development. 

The kind that lias been best i ts us up until recently is what I call the '‘Do- 
it- Yours elf" development . This ir development on the part of the local school 
district; this is developing your own pre-gram . I’m going to sound a little nega- 
tive because I don't think fait has worked out very well. Title I has sunk $G 
billies into this model cf development * and we don't have one thing to show for it. 
Maybe i f school districts have discovered a few things, but basically it's $6 
billion worth of ••xpenuive development that ho.s really come to nothing. President 
Tixon, in bio message on education, said this, 'Vo must stop lotting wishes color 
ou r judgment about the effectiveness of natty compensatory Programs, when despite 
some dramatic and encouraging exceptions, there is growing evidence that most of 
them are not was ur ably improving the success of poor children in schools.” Recent 
f-,.. lines of tno tw.j larger, t such programs arc particularly disturbing. We now 

.’-ore than $1 million per year for educational programs under Titl'J I of the 
! 1 1 •.•..lit ivy a;« i secondary Education Ac*:. Most of theca programs Iw:- stressed the 
teaohir,? of reading, Lut before and lfter tests suggest *JkU only 13 percent of 
t ‘* “ chiJL’ror; ‘f -iih program.; Imprc-v •• their reading uignii icantlv . Thi:te--r pei'- 
-t>r.t actually appear to fdl! ..chin.; more than expected, an.] irorrs than two-thirds 
the cV.lldivt: ’’v'ffl-j in UMliv-otet , that it, they continue to fall behind. In our 
iie id Start program where no a is invented, we find tint children enroll. td for 
the summer j-fuf-v 1 almost no /-ain, end the gain:; of thosr enrolled .in the program 
• or the full year arc socn alme-.t mJ * «:i.c J by their non -Head Start classmates from 
aimliarly poor backgrounds. I don 1 ’, have much hope fer local school "Lo- It -Yourself 
Yourcaif** .love lopDenr. . I chink there art- '‘.owe encouraging reasons why the pro • 
gram you are involved with at* old be an exception; i.e., that both the Hoad Start 
program and t:>e Title I monies came with strings unattached, and if you feel a 
little :;i.ovln> T , and pushing from Wash It if. ton about evaluation ’nd line items and 
some other kinds of restrict ion-. , these arc the reasons they ere there, because 
they didn't feel the "hand:,- f” gift of nt.r.oy came out very well. 

Why didn't they? What is it that you could do better to insure that this 
doesn’t happen? Let me at least talk about two reasons why I think those programs 
failed, and why you will have to make a difference if you are not going to fail. 

One is inadequate evaluation. I really don't have to talk about that; it's been 
the subject for the last two days. It is enough to say that the Title I programs 
simply haven't been able to demonstrate that they did anything that worked. 

I was a little hot under the collar about this, since we spent quite a little 
bit of money on testing. However, 3% is not very much money for evaluation. A 
project Has an obligation to demonstrate it can do something to improve things for 
kids and to demonstrate that it can be used in other schools. If this were just 
for your own local school district where you were only accountable to the local 
people, 3% might be enough. It is not going to be enough if you are going to have 
to demonstrate to the rest of the country that your project was worthwhile. This 
figure may floor you, but we spend S0% on evaluation. No program we manage in the 
Far West Lab receives anything less than about 50% of the effort for evaluation. 
Three percent is not very much; I don't think 10% is too much at all. Another rea- 
son those "Lo-it-Yourself" projects have failed is that they haven't concentrated 
enough on exportability. They haven't concentrated enough on what we have to do to 
wake it work in somebody else's school district. There are a lot of things the 
average developer doesn’t think about unless he has had the experience of trying to 
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export his project to someone else. Teacher education is certainly cne of the 
most important areas . You can go in and handle the teachers and have your own 
training session to teach theta. But what happens when the next group comes along 
and they want to do it? They can't very well go in every time the teachers want 
to do it and teach them how to do things. You've really got a separate program 
to develop, and that is the teacher education program. If they are going to be 
involved, it isn't just developing the materials the teachers use; you've got to 
develop a program that will teach them haw to use those matei'iels. So many times 
the materials are not in a form that is exportable; they are loosely bound, a piece 
here, a piece there, and it's, hard to gather these things together- and take them 
to another school district. That school district can take your idc-a, they can 
spend the same amount of money doing the same things you did. But if your program 
is really going to have payoff, they shouldn't have to go through the la burs you 
went through. That's what you were doing it for the first time, to develop the 
technology, to get the pieces together in a way that they can pick them* up . I 
think that this is the area in which your obligation as "diffusion people" becomes 
very important . 

There is another type of development happening--we call it empirically based 
development. Notice we don't use the words "research based." Research based means, 
to me, that there is a good theory, a good idea you can pick up and run with. This 
is not the case today, for we sunk a lot of money into educational research, but not: 
very much of it has the kind of practicability you need to operate programs. So you 
are going to have to do it yourself. We therefore prefer to talk about empirically 
based development, because most of it you are going to have to do for yourself and 
demonstrate for yourself. You are not going to be able to rely on a lot of research 
That kind of stuff just doesn't exist for the developer. It hasn't been until very 
recently that we have recognized the need to separate the ideas of research and de- 
velopment. 

Yesterday, in the presentations on evaluation, some of the hints of the dif- 
ferences between research and development came out in terms of evaluation. Most of 
the standard techniques the researcher knows for evaluation won't work for the de- 
veloper. He has different problems and he needs different tools. I would hope that 

with the research dollars in vocational education, one of the things Washington, D.C 
could focus on is the development of some research of how you evaluate in the field 
of development tools. You don't have the time or money to develop them yourself. 
We've got to get the guys who are complaining about the situation, who supposedly 
are responsible for evaluation, to get off their duffs and come up with some better 
tools . 

Empirically based development means you go out and test. I got up yesterday 
and made my little spiel about evaluation and about dividing up the development 
process into at least three phases. We figure that to develop one hour, one .ins true 

tional hour, well may cost anywhere from $10,000 to $25,000, and I think that may bo 

an underestimate. Divide up your money into that and you can see how few instruc- 
tional hours you would probably be able to do really well. And if you are going fo 
be this kind of developer, the thing I would urge is that you keep your project 
small, and don't try to be everything to everybody in the school district. Keep 
it small. Washington will understand. You don't always have to do everything you 
told Washington. We are pretty much adjusted to Washington's changing plans. If 



you can demonstrate why you have to change your plan, Washington will understand. 

And if it is in the interest of improving your accountability, they are flexible 
guys. There aren't a lot of formal guidelines for this kind of development. There 
are a number of organizations around that have been in this business a lot longer 
than the Far West Laboratory. They have built up quite a lot of experience, and 
of course we hired those guys to come and work for us so we could rely on their 
techniques and abilities. 

Another phase in the developmental cycle focuses on products. Products which 
are, as I said before, exportable. Let me talk just a little bit more about the 
products that work and the ones that don't. You’ve heard the name Roger bandied 
about, and I'm sure coming from vocational education, you are familiar with him 
as a researcher and developer in the fields of agriculture and sociology. 

He had a number of things to say about what it is that make? a product successful 
and I'm kind of playing with his words a little bit. I think his ideas are just 
as good today as when he developed them. First, your program used to be divisible. 
If somebody has to come in and adopt the whole thing the first time around, he is 
going to think twice about it. Fix your program in such a way that he can first 
try it out in a little way. He doesn’t have to take the whole package. Let him 
be able to first adopt components. You've got a lot of other components in the 
system you are developing, but keep it small. It’s nice to have an integrated 
system, one that works as a whole, but if you are going to have success as diffusion 
people, one of the things people will want is to be able to pick and choose from 
among your system. It can’t be complex; if it is complex and hard to understand, 
people won't buy it. 

We must learn how to advertise. We may have an obligation not to have any 
sort of hidden message, but nevertheless, there is nothing wrong with speaking 
clearly. We must develop a perfect example, and take all the garbage away and talk 
to a person in the language he wants to follow. It’s got to be communicable, it's 
got to be understandable . One of the ways you can make it communicable is to make 
it Visible. If it's going to be guidelines, dress it all up in a little booklet 
and call it a manual. Don't leave it in a little sheet like this one I'm going to 
pass out to you. Dress it all up a little better. Mine will never get used and I 
know it won't, for I didn’t have, time to dress it up. But if your products are 
going to be used, you are going vo have to put them in a form people can see, feel, 
touch and smell. So I therefore urge you to keep them in a product form. It can 
be a process, yo\i can package processes very easily. But it's better if it can be 
visible and if it can be demonstrated in some way. It's got to be compatible with 
what people are now doing. This is hard, for you want to change things and that’s 
what this program is all about. But if you change everything, people are going to 
resist. Find out what they are doing now, and say "O.K. , how can I take the fact 
that they do it this way and play on that fact and turn it into something else?" 

You can change them but give it to them in a way that they are now accustomed 
to using. 



Last, but not least, to make your developed product as exportable as possible, 
you have to show the relative advantages of your pi’oduct . People must be convinced 
that what you have to give them is better than someone else’s product. The farmers 
did not adopt hybrid seed corn until they could see the increased production be- 
cause of it. People must see why a product is better. I think educators are pretty 
smart. They know what works and what doesn't work, and they can see through someone's 



fancy research and these demonstrations and they know whether it will do a better 
job for them or not. I think the very fact that you all are out there developing 
programs suggests that you've got some kind of estimate as to whether they will 
work better or not. This doesn’t mean that you can't add to that judgement. But 
unless you can show that it is going to be something substantially better, you 
are not going to have much of a chance of having it widely adopted. 

Let's suppose now that you are this ideal kind of developer and you spent 
your $25,000 for an instructional hour and formed a project and you've got it all 
ready to disseminate. How are you going to diffuse? That’s really the thing I 
want to talk about this afternoon. There are lots of channels. One of the ways 
I haven't heard anyone talking about (but I don't see why it won't work) is through 
commercial channels. Why can’t you clean it up and sell it to a publisher? You 
can make a little off of it, the school can make a little bit off of it, and it 
will be widely distributed because the publishers are very good at seeing that 
things are adopted. So the commercial channel, in my judgment, is a very useable 
and a very realistic route to go if you want to see that diffusion occurs. If you 
do go that route, I would like to make a few suggestions. U.S.O.E. has really had 
a fantastic turnabout in their attitudes toward copyrighting. It used to be that 
you couldn't get a copyright on anything that was done under a Federal grant. The 
theory was that you put it in the public domain so that everybody could use it. 

That was good in theory but it just wasn’t very good in practice. The publishers 
must have that copyright to insure that someone else doesn't come along and take 
the same idea. They've got to be fairly sure they can spend the money, and it does 
take quite a little bit of money on the part of the publisher to get this thing all 
packaged up. They've got to be able to protect that investment. You get the copy- 
right and in a sense you assign that copyright to the publisher. So if you do in- 
tend to go this route, one of the things you will have to do fairly early this year 
is to notify your contract officer of your intent. You can get a developmental 
copyright to protect your material during the process of development so that some- 
one doesn’t lift it. More than protection from lifting, such a copyright will in- 
sure that if it isn't any good, it doesn't get diffused too widely. This is some- 
thing you will have to work out with your contract officer. If you are going to 
go that route, you will have to worry about the clearances for your material. Most 
of the projects I've heard about are going to pick up a little bit here and a little 
bit there. If you start picking, and you pick up someone's copyrighted material, 
you are never going to be able to interest a publisher in what you have because he 
can't deal with all those tangled alliances with other copyright holders. This is 
a whole big mess, and you are going to have to begin thinking about it now. 

The nonprofit-making channel is to utilize the existing apparatus in U.S.O.E . 
The Government Printing Office is certainly a channel you can go through. Some- 
times it takes an arm and a leg to do something, but if you can get your contract 
officer to exert that kind of pressure, they will publish your materials. You can 
also sort of sell it yourself, that is, sell it through ERIC. I’m not going to 
urge that route on you. My personal feeling is that if we are going to have pro- 
grams widely distributed and adopted, we have to go through a commercial channel. 
There is another facet to diffusion along with a choice you have to make. What kind 
of media are you going to use? I heard some of you talking about television. 1 
don't know if you know anything about TV or not, but the technical hassles, and the 
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compatibility among tape recorders are just - wow*. If you are going to try to 
distribute anything via videotape, you are going to have your' hands full because 
no one's machine will take the tape from anyone else's machine. Someone fortu- 
nately came up with an audiotape which is somewhat compatible. However, except 
for large commercial machines, almost no one's machine is compatible with anyone 
else's* If you are going to go this route, you will probably have to use film, 
occause everyone seems to have 16mm film projectors. And there, of course, you 
have another problem because if you put the material originally on videotape and 
then try to transfer to film, you've got a black and white product of inferior 
quality. Your choice of media, particularly if you are going to go the AV route, 
is going to be difficult. Don't let any of these media salesmen come in and sell 
you some fancy gadget that coordinates filmstrips, etc. They have them, but no 
one else has them. You can sell it to one guy because he has the machine but no 
one else has it. So keep within the standard 16mm film, 35mm slide range. At 
least half the schools do not yet have the 8mm super projectors. So you see you 
can’t even go that way if you expect to get wide distribution. You also have 
problems when you get to facing your decisions about how you are going to diffuse 
your project or whether you are going to have one-way or two-way communications. 

The easiest way is, of course, one-way , and simply send it to the guy. But this 
morning I think your common sense tells you there are better ways. You can have 
conferences, you can have people come in and look at your project. This is one of 
the ways U.S.O.E. has been handling diffusion-dissemination. They set up one of 
these model projects and then have people come visit it. The people who come visit 
get a lot out of it, but what about all those other people who didn’t or couldn't 
come? If we are really going to get an effective diffusion exchange in the educa- 
tional system, we are going to have to get to everyone in that system. So, you've 
got to decide whether your diffusion channel is going to be one-way or two-way, and 
all I can do is urge you to think about the most effective choice you can possibly 
make . 



Let me show you how we are handling one possible model which can be used for 
this purpose. The model I'm going to demonstrate to you is through a commercial 
channel. It's a multi-media package and it's a one-way package. It's simply a 
box, this box. This was our first effort at disseminating information about an 
exemplary program. We started about two years ago in the field of elementary 
science. Science has been lucky. They've received lots of money for quite a while 
from NSF. It's a little easier field with which to work. There were six elemen- 
tary science projects that had been funded to the tune of $10 million. So what we 
did was package information tools to describe to educators what they could adopt 
that was exemplary in the field of elementary science based upon the exemplary 
programs which had been funded by NSF. The package has the following characteris- 
tics, and these are characteristics I think you should look to when you are involved 
in developing your dissemination product. First, it's got quality control. We 
didn't want every Tom, Dick, and Harry's program to get into it. We had some very 
specific requirements that only certain kinds of programs could get in here. We 
have what we call an educational development identification form. I will give you 
a copy. We are now in the process of mailing out 2,000 of these forms to educational 
developers. We are rating these developers. I thought it might be interesting for 
you to get an idea of the kind of questions we asked. Not every project has every 
one of these features. But they have to have enough of these features to begin to 
look like a development which we think has a chance of surviving in the schools . 
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The objectives of the development project are described rigorously. They use per- 
formance objectives. A full-scale feasibility study using a sample target audience 
of the proposed project must be carried out before the initiation of the proposed 
development. The materials and procedures developed by the project must be tested 
under control situations which ensure that the conditions prescribed are proper- 
and that the conditions were followed. The material and procedures developed for 
the project must be tested in settings which the developer did not control and where 
the user was free to use the product in any manner he saw fit. The product or pro- 
cedure resulting from the project must have undergone successive recylcing. 

I think this is one of the most encouraging things I have heard about your pro- 
posals. You have three years and you can recycle and change your pi'oject the second 
and third years to make it better. It's impossible for you to get it right the first 
time. You must have a couple of trys at it before you can get it fight. 

Quality control is one feature on which we do try to insist. It's multi-level. 
Someone said this morning that a superintendent or a principal does not have time to 
read a thirty-page technical report. In this project, we have three levels of in- 
formation. There is what we call a summary level of information, starting essen- 
tially with things like these charts that show comparative differences among pro- 
grams. A chart is a fairly simple way to communicate to a fairly high level person. 
He can spend just a few minutes and get the main message (or messages) out of what 
he is reading. We also hate short summaries, no more than one hundred words, so 
that some kind of an easy, quick run-through of the project can be handled in five 
minutes. If it doesn’t interest the reader in that five minutes, he will put it 
aside and not look at it any more. At a second level, we have what we call audio- 
visual filmstrips. These are descriptions which take about twenty minutes. They 
show what the program looks like in action and they show what the material looks 
like. It is very much Tocused on what happens in the classroom. In our little box 
here, we have a one day cimmary for each of the six projects, a chart that compares 
them , and audio-visual filmstrips which are particularly suited for a little more 
intensive program evaluation but nevertheless not really the kind of thing you get 
in a written report. I think you are going to find yourself confronted with a lot 
of visitors. You may want to rely upon this means of two-level dissemination where 
you can give the visitor a one page handout by which he can get a picture in his 
mind before you start briefing him. You can get a great deal said on a fifteen- 
minute filmstrip and you don't have to show the visitor everything. 

In this particular package we also have a report or a description of each of 
the programs. These reports are developed by the Laboratory* we wrote them for the 
projects themselves. We wrote them according to a specificaion I will pass out to 
you. It concentrates on the nitty-gritty of the classroom activity. Last, we also 
have enclosed a history of the project. We put the history last in our system. 

They do like to have it eventually, but only when they get around to it. 

Even if you can’t do this filmstrip thing and all the other things, when you 
are writing your final report write a one page abstract which will be the first page 
in the report. Then write a three or four page summary of the whole project and in- 
sert it behind the abstract. Don’t put it in last, put it in first. Then put the 
technical report behind the summary. The reader will really appreciate this. Try 
to have the information available at several levels of detail. 
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I've already showed you a multi-media approach to diffusion and I think there 
is a more satisfactory one, particularly when you are working with groups. For ex* 
ample , we now are developing packages in the area of elementary science., secondary 
social studies, early childhood education, individualized instruction, and we are 
hoping to focus next year on ecological environmental education. Not all these are 
multi-media, but they ai*e the ones which are the most successful. We were working 
with a field test group on the early childhood project the other day and most of 
the people at the consumer end of early childhood education speak a second language. 
It's not English, so in our early childhood education program there is a bi-lingual 
system built into the information. If you nave a vocational education project in 
an area where bi-linguality is a feature of the area, I recommend strongly that 
your material be in two languages . 

One of the features of this system is that the characteristics of this box we 
are using is that it is mailabue. Again, this represents the one-way link in commu- 
nications rather than the two-way link. We decided that if we were really going to 
spread the word about six new science programs to 20,000 school districts, we were 
never going to do it if we had to go to every school district* The only way we 
could get it there was to go through the mails. That's the reason we have film- 
strips. If we had to have slides and a big carousel, it would never go in the mail. 

I have heard proposals about how the educational field might disseminate in- 
formation about their exemplary programs and vocational education. I would like to 
present an alternative to these previous models. I don't want to appear to be 
knocking other models such as ERIC. I think it has its role, particularly within 
the research domain and between the research and development domain. There is no 
place else you can go now to find out what is going on in research. However, I 
don't really think ERIC is going to solve the problem with regard to getting the 
information from you to the schools . When you look at who uses ERIC and how it 
functions and how it is organized, I just don't think it is going to do this job. 

I think there are some other things it could do. 

Just suppose you are all sitting back at your desks in your offices and sup- 
pose you are not running an exemplary project. Suppose you had on your desk a 
little box - this is called Granny's recipe box - it's just a little 3x5 or 4x6 
card index, and in this little box are cards. Each of these cards represent one 
of the 250 best exemplar 4 / progx'ams in vocational education. Each card includes a 
project name and a brief description. Let's also suppose that around the edge of 
this card there were little punctures keyed to a manual you have to describe these 
holes. These holes represent variables in the exemplary programs - type of project, 
grade level of project, etc. In this manual you might have a little card attached 
which you could send off to get more complete information about the program with 
a detailed evaluation report on the project. Because you are sending for the in- 
formation based on your needs, you are likely to get information that is most likely 
to be of use to you and you don't have to go through the rest of the cards. 

Let's suppose you could get a box like this and it could be individually 
tailored to your needs. The system isn't very complicated and it wouldn’t cost a 
lot of money to develop if it was done on a nationwide basis. You can't do this 
yourselves. I don't really believe that the state educational agencies have the 
kind of money or resources to do it by themselves. What I am proposing is that 
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you could convince the vocational education branch of U.S.O.E. that 1% of the 
money they put into exemplary projects ought to be spent in developing an infor- 
mation system so that at all times you have seme thing you can look at. I think 
a system like this could be developed; I think it could be operational. 

Incidentally, when we field tested this box, we found that since this box 
was oriented to decision making in order to help middle level school management 
people make decisions about what they want to do to change their elementary sci- 
ence program, 60% said they could get all they needed to know to make the adop- 
tion decision out of that box. Ninety percent said that it was a much more ef- 
fective way than paying $150 to a consultant to come in and then find that they 
still didn't know anything about elementary science education. Therefore, J think 
this type of a model can and possibly would work for disseminating information on 
a vocational educational program. 

Let me stop with one more little comment. One of the things which make people 
nervous about this box is that someone else wrote it. It's sort of like hiring a 
third party to do your evaluation. As an independent agency working with a number 
of other agencies whose products were represented, we were suspect in the same way 
you are going to be suspect to the independent evaluators. Again, with pride I 
want to say that we have had more compliments about how well we described the 
project from the project people themselves than we could ever have hoped to have. 

I think it is just because it helps to have someone from the outside come in and 
say "this is important," "that is not important," "people want to know that," "tell 
them this or that." We are not science people and we are not social studies people 
Our job is communication, and I honestly believe it helps to be just a little bit 
stupid in our role. Because our heads are pretty thick, you've got to explain a 
thing pretty well if we are going to understand. If you can explain it to us, I 
think, then, we can do a good job of describing it to other school people. 

I propose this to you as a model. I do so recognizing that I seem to be im- 

plying that we would like to do it. I don't really think that we have to do it. 
There are a lot of good agencies around that could do this very well. 

Let me close with a little story of a farmer who had just hired a new man. 

The first day he sent the man out to cut some weeds along the fenceline and he fig- 
ures it's pretty much of an all day job. He came back about noon to check on the 
man and the work was done. Well the next day, the farmer had a bunch of wood to 
chop and so he sent the man chopping wood and again he thought he had given him 

enough work to keep him busy all day. The farmer came along about 10:30, and the 

work was all done. The third day, the farmer thought, "This guy is great, how 
lucky can I be! I don't want to push my limit so I'll give him an easy job today 
and I'll not even come back until 4 o'clock so if he wants to sleep a little bit, 
he can." So, the farmer sent the man down into the cellar to sort potatoes and he 
had to put the good ones over here and the bad ones over there. So about 4:30 the 
farmer comes back and here the guy was, still sorting potatoes. How come? Why? 

The hired hand said, "Decision making, that’s tough!" 

Well, you have some really tough decisions to make and I think it is going to 
take more time than you realize. Good luck in your endeavors. 
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You have been a very patient, dedicated group. You have stayed with the 
Institute program right to the end. I've heard some very fine presentations thus 
far. It will be a challenge indeed to come up to the standards set. 

Thus far, your concerns have focused on what might be called tactical prob- 
lems in administration. This includes processing kinds of information that might 
be generated during the project, coping with the excitement and interest that may 
come along as a result of being involved in an exemplary project, and deciding how 
you might diffuse findings related to your projects. I should like to speak in a 
much broader context. At the same time, it is recognized that project administra- 
tion or program management represents a kind of a microcosm, a subset of the to- 
tal that is known as administration of activity. I submit that administration is 
a means to an end. It can have rather significant and profound impact on the suc- 
cess of the project; as much as the substantive content or the quality and validity 
of the research behind it or the design selected for the project. Projects can 
succeed or fail because of a lack of administrative capabilities. Administration, 
then, is not an incidental thing to an exemplary project. Rather it is something 
vital to the totality known as school administration. 

What is called project administration, or program management, is relatively 
new in education. As a matter of fact, if you were to search the ERIC descriptions 
for references to project administration, I suspect that you would pretty much come 
up with little to read. There may be one or two things, but I would be surprised 
if more existed. About a year ago, AASA came out with a publication called Admin- 
istrative Technology . We almost devoted some pages to project management. It was 
to be introduced at the tail end of a chapter I was writing on program budgeting, 
PPBS. The decision was made that it really wasn't intimately related with PPBS . 
There were some similarities . One could adapt some of the concepts to PPBS or sys- 
tems administration in general. 

In contrast to the small amount of project management in education, a consid- 
erable body of literature on the topic exists in industry. Special projects are 
used to develop an unusual or innovative product. Dupont had a special project 
staff which probably launched nylon and corf am. In the military, it's quite common 
to find new weapons systems being developed via special project teams- Thus, there 
was the Polaris missile project, the Nike-Zeus project, etc. 
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Project administration, or program management, is relatively common in the 
field of business as well as in the military. We find that it is a particularly 
useful device for the development of new approaches. And I submit that project 
administration may benefit education just as much. Within the next five to ten 
years we will discover more concern, a greater body of literature dedicated to 
the unique problem confronting people such as yourselves, dealing with Exemplary 
projects in the field of education. As the administrator is conceptualized more 
in terns of being a change, agent than simply the efficient maintainer of the sta- 
tus quo, which appears to predominate much of our present writing, then there 
will be indeed greater emphasis on project management. We might well follow the 
precedent set in the development of new weapons systems, space programs or special 
kinds of programs for the disadvantaged, in government as well as in business. 

Project management is concerned specifically with the introduction of rapidly 
advancing technology or research into the school district in the shortest period 
of time possible, with the most efficient utilization of resources and with a min- 
imum amount of dysfunction within the system. Project management can be viewed as 
a special set of administrative processes or techniques that help to effectuate 
change with a minimum of dysfunction. 

In the literature that exists in the field of industry and government, parti- 
cularly in the military aspects of government, project or program management is 
sometimes referred to as systems management. Now in most fields, and I would sub- 
mit that this is true as well in education, systems management is a generic term 
which suggests a high degree of reliance on the systems approach. I will refer to 
the systems approach later on because it is closely related to project administra- 
tion. 

Project management concepts evolved from systems engineering approaches used 
in complex industrial and military production problems. This product-mission con- 
cept underlies operations and is related to what has been called PPB systems. A 
planning, programming, budgeting system which sane call program budgeting. My own 
feeling is that PPBS encompasses a sound idea, but the term PPBS is a poor state- 
ment of the ideas. I've been critical of those who insist on calling the system 
PPBS , or program budgeting , because it is contrary to the mission orientation of 
the new system. A far more precise term is RADS, Resource Allocation Decisions 
System. RADS puts the emphasis on decisions to allocate limited resources to a 
stated mission. A special project is reviewed in terms of its missions with plan- 
ning and programming evolving in subsequent steps. 

The modifier "exemplary" in the term "exemplary program" implies uniqueness 
or that the program is full of promise. But the key concept is program. We have 
a bit of a hang-up in administration as to what constitutes or how one defines a 
program. Within the next decade, we will evolve a programmatic classification sys- 
tem for all of education. Right now, your projects are unique in that they repre- 
sent a program that is somewhat separate and distinct from the typical ways of or- 
ganizing education. It is a great challenge. It will take at least a decade for 
this challenge to be realized. 
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It can be defined as any cluster of activities related in a significant way 
to achievement of a stated mission. 

An exemplary project may be perceived as a set of activities which are re- 
lated in an intimate way tc the achievement of a mission or a given set of objec- 
tives. The success of an exemplary project is measured by the degree the parti- 
cular goal was attained. Again please note the very close relationship between 
the program definition and accountability. Accountability is basically a politi- 
cal term. It is a term which has clearly deposed relevance as being the "in M 
term. Two years ago, we talked about all things being relevant. How, we are in 
the period where the word accountability pops up instead. It scorns to stand by 
itself. This is suspect because accountability means that you are accountable for 
some thing. The implication is that everybody understands that accountability is 
related to results. What the term accountability achieves is relatively unique 
in that it switches the focus from simple itemization of the input, or a descrip- 
tion of the process, to results, the end product, or the output achieved. 

Accountability is a political term whereas productivity is an economic term 
and assessment is an educational term. At least the better term is more familiar 
and has been used for a more extended period of time in education. In effect, all 
are output oriented and are somewhat related to what we call the program approach 
where one speaks of a math program, science program, music program, art program, 
and even vocational program. This has been the traditional way of describing and 
organizing educational institutions. There are some in the PPBS field who tend 
to stress the idea that PPBS is a kind of a cost accounting system, that is, it 
helps you know how much was spent for science in the elementary school or in the 
secondary school. I would submit that this is not related in an intimate way to 
the program definition presented herein. To program in the sense suggested herein 
is to think in terms of missions to be accomplished rather than the processes of 
instruction in such fields as science, mathematics, or vocational education. Again 
it is more in terms of the output than in the number of people assigned. 

There is a need to appoint someone, such as those present, as project managers 
with overall responsibilities for planning, for operation, and ultimate outcomes of 
the project. More than likely, the position of the project manager will be some- 
what unique. It may be superimposed upon the existing formal » and functional, or- 
ganization. We can justify this on the grounds that what we are attempting to bring 
about is a significant change in the system. It can be argued that the framework 
that has persisted thus far is not the best framework for creating change. In pro- 
gram management there is a call for organizational modification. This does create 
a new and more complex set of organizational relationships. And herein lies one of 
the big worries, in my estimation, for directors of exemplary projects. There are 
relationships within the project itself demanding leadership in organizing, in 
staffing, and in coordinating. The relationships established between the project 
and the rest of the system are crucial. The project is a subset of the total sys- 
tem. A clear definition of the place of this project in the overall educational 
operation is a must if any degree of success is possible at the termination of the 
project. To whom does the project director report and can the project director cut 
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across existing administrative relationships or hierarchical relationships within 
the total system? These are crucial questions. The development of a new weapons 
system in the military, or the operation of a space program, makes it necessary to 
cut across existing organizational relationships, lines of authority so to speak, 
to achieve this goal. These operational prerogatives may irritate some people. 

There are vested interests as to who has authority to speak to the chief executive 
on a particular issue. Some may argue that if you look at the formal chart of or- 
ganization in the school system, the project manager should report to a given ad- 
ministration and to no others. Host project managers report directly to the chief 
executive of the department in which the project is located, if not to the chief 
executive of the entire system. If I were a project manager, one of the very first 
things I would insist upon would be that there be a clear definition of reporting 
responsibilities and relationships. This is important for reasons other than ag- 
grandizement. The issue is not that you are trying to build an empire, but rather 
of recognizing the intricate human relationships and the petty jealousies which 
could be generated if these were ignored. It will be extremely difficult for a 
project manager to fulfill responsibilities laid at his doorstep without a clear 
definition of authority as coordinator or expediter within the system to cut 
through the usual subordinate lines of authority. This may help to avoid conflict 
with the functional administrators, such as principals, directors, deans or depart- 
ment heads, while striving to realize the goals of the project. 

You must define the boundaries of operation very carefully: You must specify 

to whom the project manager reports. You must have a clear understanding of what 
kinds of resources, and their magnitude, are available to the project. You must 
he aware of the freedom of movement, so to speak, within the system. 

The project manager’s task usually has a definite time limit unless it is re- 
vised at a subsequent point in time, such as two, three, or four years. He is usu- 
ally given set resources upon which to draw. If there is a need for more resources > 
then clearly some kind of major budget alteration must be made which entails special 
and formally granted permission. 

When the project is completed, it is assumed that the project manager is as- 
signed to other activities. There are occasions, and sometimes this is not clear, 
where the individual is employed simply for the project and there is no guarantee 
that he has any tenure in other parts of the system once the project is terminated. 
Unfortunately, those who do the employing are somewhat fuzzy in specifying tenure 
at the time of hiring the project manager. In turn, the project manager, while 
employing other personnel involved in the project, may be unable to clarify the 
status of such personnel within the organization. Thus, a secretary may obtain the 
impression that she is being hired by the school system and is a regular employee 
of that system, no matter what happens to the project. The assistant project direc- 
tor or the specialists who are involved may have similar feelings. The failure to 
specify the limits of employment may generate problems later on. And I think you 
will discover, as frankly I discovered in the launching of a new project within 
the American Association of School Administrators, that you don't always think of 
every little detail when you start an operation. 
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Let me relate, as a point of illustration, from my own experience in project 
administration. We employed an associate director for the project. He was given 
the title of Associate Director for the NASE. We failed to define his relation- 
ship to the AASA in general at that time. Now, my initial position in AASA was 
as Associate Secretary (in addition to the Academy project, I served as secretary 
of the Ethics Committee, secretary of the Committee fcr Advancement of School 
Administration, etc.) So I wore several hats and my base in AASA was clear. Your 
position in the system may be clear if you were employed prior to your appointment 
as project manager. I assumed that an associate of mine in the Academy would bear 
a similar relationship to AASA but I was wrong in that assumption. At the time of 
employment, he should have clarified whether an Associate Director of the National 
Academy was also an Assistant Secretary of AASA. Then his position in the total 
organization would have been clear, and a greater degree of permanence on the staff 
would have been recognized even if the project was terminated. 

I believe there is a trend toward greater utilization of micro-administration, 
that is, an increasing importance and greater utilization of program managers. 
Project administration is likely to grow rather than diminish in public education. 

My position is based on the assumption that the emphasis on innovation will grow 
rather than recede in the days ahead. I believe there will be more exemplary 
projects rather than fewer. There will be a larger number of projects and pilot 
studies placed in a natural school setting involving teachers, classrooms, students, 
and buildings under the jurisdiction of a functional administrator, such as a prin- 
cipal, a director of a vocational technical school, a dean of a college of educa- 
tion, or the chief of a department in a state department of education. 

Project administration may be conceived in terms of specified processes. Before 
moving into Lr'is, of course, there is a need for definition of the project mission. 

I am assuming that this will have been done or the exemplary project would not have 
been created in the first place. Then comes the project plan. This is extremely 
important. X suspect that very often administration gets so involved in operation, 
in putting out the fires, that the aspects of project administration most likely to 
be neglected are project planning and evaluation of the planning. This is true of 
long-range planning for the entire period of the project, as well as for short-range 
planning, that which is to be accomplished within a given fiscal period or a part of 
a fiscal period. Assessment to determine whether or not the plans were achieved is 
neglected more often than not. This is intimately related to project control. Con- 
trol is considered by some to be a dirty word because it implies dictatorial behav- 
ior. I will submit that unless the project administrator develops an early warning 
system which will tell him (far- enough in advance so that he can begin corrections) 
that the project is falling way behind schedule; or that there are inter-personal 
difficulties which are generating a potential crisis; or that the project is in 
serious financial trouble, the probability of success will be diminished. The 
project administrator needs an intelligence system, if you will, to provide the 
kinds of information input, or a feedback loop to allow him to interpret the state 
of affairs. He may discover difficulties only after it is much too late, or when 
the project blows up completely. 
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The thermostat is a mechanism which senses the environment and within about 
a 2-degree range can sot into motion corrective activities. We should have the 
capability to do the same and I am suggesting that as project director, you must 
be concerned with the control elements ; not in the sense of dictating what shall 
be done, but in the sense of understanding the rhythm of the project. This means 
being able to detect whether you are indeed locked onto the target or whether you 
are moving in an unproductive direction, however small the error may be, that will 
result in being wide of the target. 

Time will permit reviewing only one of the planning devices called the sys- 
tems approach. More and more people are talking about program evaluation and review 
techniques (PERT). It is hardly new. It came out of a project environment so 
far as one of the well-known applications in education is concerned. The work of 
Des Cock, I believe, in testing and evaluation at Ohio State, led him to be in- 
terested in a particular systems technique which enabled him as a project director 
to have a better understanding of hovr best to organise his project, how to set a 
time constraint for each aspect of the project, and how to evaluate and thereby 
control whether the project was off or on target:. I'll speak just briefly about 
the PERT approach. 

Planning , both long- and short-range, is extremely important. I know of no 
football coach who would go into a game without a kind of a game plan that would 
specify the strengths of his team and the weaknesses of his opposition. He ar- 
ranges his series of plays or activities on the field to exploit the virtues of 
his team and the weaknesses of the opponents so that his team might achieve the 
goal of victoxy . Obviously, every project manager must be concerned with organi- 
zational "game plans." I will talk about this only in terms of the people in- 
volved, that is the leadership concern. The plans for ccordination, for super- 
vision and control, are part of the leadership concerns. Then clearly you have a 
responsibility to know for what you are going to be held accountable. This is a 
thumbnail sketch of project management, recognizing the constraints upon my time. 

As a project manager, you will have to understand what kind of auditing is to 
be expected — auditing of the way you have utilized the fiscal resources placed 
at your disposal. You must know whether you have authority to spend money for cer- 
tain things, and at what rate, and what kinds of purchases are prohibited. It may 
be as mundane as specifying that there is a district policy of paying 10 cents per 
mile for the use of the car. You may conclude that it costs more to operate a car 
and the district poliev is wrong; so as project administrator, you decree that trav- 
el payments will be based on 12 cents per mile. Whether you know it or not, you 
will be held accountable for this switch in policy at the time of audit and the ex- 
tra 2 cents per mile may be disallowed. What kinds of invoices you are going to 
i.ave to submit as evidence of payments , what procedures you will have to follow 
for such pedestrian activities as purchasing, and what kind of inventory records 
will have to be maintained should concern you now as you start the project. Fed- 
eral auditors come in sometimes two or three years after the termination of a 
project. They have been known to disallow Federal expenditures made outside the 
requirements set at the time the project started. I’ve heard of more than one 
school system using Title I being forced to return six to eight thousand dollars 
because they were not mindful of the Federal auditor's demands. There is an ex- 
treme position in all this. I accused the business manager in a school district 
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of being bitten by a Federal auditor at a very early age. The business manager 
was very rigid and demanded more than was desirable and necessary. This extreme 
can stifle and make very uncomfortable the project environment. 

The point is that the care exercised at this point in time, and during the 
project itself, may be the thing that saves you from considerable embarrassment 
at the end of the project. Even though you are responsible for the substantive 
content of the project, that is, carrying through the given activity related to 
improvement of some aspect of vocational education, you still need to know enough 
about the business management end that it doesn’t come back to haunt you. As mun- 
dane as business management may be, it is something which you cannot ignore if you 
hope to preserve your reputation as an effective administrator. The project man- 
ager must sharpen his administrative skills. Time permits the illustration of 
only a few systems techniques which have been evolved in administration. It is 
recommended that project administrators develop such skills. The National Academy 
offers seminars in Advanced PPS and Systems Analysis. Even in the intensive, one 
full week not all concepts can be covered adequately. 

A system can be defined simply as any group of two or more people who have a 
goal and a plan for reaching that goal. An exemplary project is a system. If the 
boundaries of the system alio*/ interaction with the surrounding environment, it 
is known as an open system. If there is to be a meaningful exchange between your 
exemplary project , as a system, and the environment surrounding it, then plans and 
organization must be made to achieve this particular goal. It will not happen nat- 
urally. Open systems are far more stable, more likely to change, and more likely 
to succeed. Closed systems, by virtue of the fact that they receive no inputs from 
the environment, are unmindful of the magnitude of the surround forces and may be 
destroyed by pent-up pressures. 

The cluster of operational behavior, administrative activities, that identify 
a systems oriented administrator or project management would include these salient 
features : 

1 . There is a delineation of long- and short-range objectives capable of be- 
ing translated into operationally meaningful activity and subsequent evaluation. 

2. There is a recognition of the dynamic nature of goals and an ability to 
sense when new ones have emerged within the project, or when a recording of prior- 
ities among existing objectives is imperative. To succeed lere , a project admin- 
istrator must schedule and protect his time for planning activities and he must 
develop a staff large enough to enable him to do this. 

(As an aside before listing the rest of system characteristics, it is well to 
recognize that the day of the one-man superintendent is pretty well gone by the 
board. Administrative teams administer most school systems. This team includes 
a cluster of assistant superintendents, principals, supervisors, and so on down the 
line. No one man can get the job done. I think that project managers would be re- 
peating a mistake in the history of school administration if they persisted in be- 
lieving that they have all of the specialized talent needed to be effective as ad- 
ministrators of exemplary projects. Now you may not have an operation large enough 
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to command full-time assistants. T'm not arguing for that. I'm not arguing for 
the, creation of a massive bureaucracy to discharge a $50,000 or $100 ,000 project. 
There is another alternative to a development of a bureaucracy. That is to employ 
part-time consultant help, either outside the system or within the system. These 
people can be assigned on a part-time basis and funds may be dedicated to this pur- 
pose . 



One of the characteristics of the systems approach is the interdisciplinary 
approach to problems. Let me go back again in history. Systems, or as it was 
then called, "operations research," really had its beginning around the time of 
World War II, not in the United States but in Britain. They relied heavily on 
this approach. Radar was a lab: .-atory curiosity. A scientist discovered that 
when a radio wave of a given frequency strikes a solid object it bounces back. 

It can be seen as an image on a radar screen. It took scientists, engineers, 
and military experts working together to develop the military and airplane use of 
radar. Here was a teaming of a number of specialists to achieve a common goal. 

Likewise, when German U-boats threatened to cripple the flow of supplies 
from the United States to Britain, it was the operations analysis, or the systems 
approach, to administration that developed a pattern of deploying ships to mini- 
mize the destructive effects of the submarines traveling in wolf packs. It was 
the systems approach as well which led to tripling the number of U-boat kills by 
aircraft. How did they do that? Traditionally, according to the military experts, 
the maximum explosive force for a depth charge was at one hundred feet below the 
surface of the water. All depth charges were set to explode at that level. Ob- 
viously, the submarine had to be at that point to feel this maximum explosive 
force. Mathematical analysis showed that when a plane sighted a submarine, it was 
assumed that the submarine sighted the plane as well. 

The submarine would immediately go into a crash dive. The airplane would 
move into an altitude for the immediate dropping of depth bombs . A review of all 
the time factors involved demonstrated that the submarine simply could not be at 
a depth any greater than twenty-five feet below the surface of the water. But 
the charge was set to explode at one-hundred feet! So, the fuses had to be rede- 
signed to explode at twenty-five feet. When the new fuses were used, the results 
showed that they tripled the number of U-boat kills. Again note the team approach, 
an inter-disciplinary approach. I think a project administrator must recognize 
that there are such specialized talents and he will have to bring them in to make 
the program operate . I go back, now, to systems characteristics.) 

3. Recognition of change as normal. Organizations really can't go back to 
the "good old days" where all was stable and little changed. 

4. The generation of alternatives means utilizing resources to reach objec- 
tives. What makes your project exemplary (and makes you a qualified director of 
an exemplary project), is your willingness to maintain this degree of flexibility 
necessary to switch to new means in order to utilize your resources and obtain 
objectives. After all, it’s the objective, rather than the means of the process, 
that is important. 
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5. The creation of models to study all or part of the system. Models have 
been talked about in this program — models of a dissemination system and of an 
informative system. I speak of a model of project management. By a model, 1 
simply mean clear identification of the key factor, and sensing the relationships 
or patterns existing among the factors. 

6. The utilization of quantitatively oriented tools, vehicles, and proce- 
dures in the analysis of systems. Here again, I doubt whether there is a fraction 
of 1% of superintendents of schools, much less project managers, who can handle 
linear programming. I'm not suggesting that project managers have mathematical 
skills. They ought to know the advantages of mathematical analysis and go out on 
the market and buy such skills. 

7. Identification of high priority of the time schedule of the top echelon 
to administrative and planning activities. I’ve talked about the interdiscipli- 
nary team and a consideration of coordination of the ever growing number of spe- 
cialists within your project as a matter of high echelon concern. It is entirely 
possible that specialists within a given exemplary project will go off by them- 
selves and think that they are important as a group, unrelated to the project it- 
r.elf . 

8. Lastly, recognizing the importance of an implementation of sophisticated 
objectives and scientifically oriented procedures in decision making. 

A host of techniques have been developed for systems administration. Time 
precludes talking about PERT, which I have already mentioned. There are a host 
of others as well. But what I am saying is that you literally are in the initial 
wave as far as the administration of education is concerned. I think we are going 
to see more of project administration in education, rather than less. I am saying 
that you have the additional task of not only knowing what your project is all about 
in a substantive way, but also of developing concepts and skills that will make you 
an efficient administrator of your project as well. 
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Expectations for Exemplary Project Directors 
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Pilot and Demonstration Branch. U.S.O.E, 



Dr. Sidney C. High 

Senior Program Officer 
Pilot and Demonstration Branch 
Division of Vocational and Technical Education, U.S.O.E. 



In this paper I would like to highlight some broad expectations, or more 
specifically, what is expected of the sale of Part D of the Vocational Education 
Amendments of 1968. Why did Congress put it in the Act? And what are exemplary 
projects across the nation expected to accomplish? 

If one looks back to the Vocational Education Act of 1963, he will recall 
that for the first time Congress put in a provision for vocational research. 
Section 4-c of the 1963 Act stated that 10 percent of the appropriation for vo- 
cational education should be reserved by the U. S. Commissioner of Education 
for research and development. I think that it is safe to say that this was the 
first time in history that a sizeable amount of money was especially earmarked 
for this purpose. Section 4-c was first funded in Fiscal Year 1965 and it con- 
tinued in operation for five years, during which period approximately $64 mil- 
lion were invested in vocational research. That’s more research in the voca- 
tional area than has been done in the whole previous history of the Republic. 

The result was, of course, a very intensive five-year period of vocational re- 
search and development. 

As Congress looked at these five years of activity, they formed the impres- 
sion that a great deal of the research and development had been carefully done 
and well conceived. However, this excellent research had gone ontc microfiche 
and had been put into the ERIC system, and therB it remained. And so, Congress- 
men were concerned about what could be done in order to take some of this exten- 
sive and expensive research and development work off the shelf, out of the ERIC 
system, and into the local school districts where it might be put to use in im- 
proving vocational education. 

At that point, Congress conceived in the Vocational Education Amendments of 
1968 the idea of Part D for "exemplary projects." What they were saying, in ef- 
fect, in Part D was "Here is some Federal start-up money to help those, who, dur- 
ing the next three years, are willing to take some of the research and development 
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out of the ERIC collection and bring various components together in a way that 
will be useful in operating a better vocational .rogram in a local district.*’ Fur- 
ther, it has been assumed that after three years, when the heavy start-up costs 
are over, and the maintenance costs are beginning to level off on a continuing ba- 
sis, then state and local resources will be able to continue these new activities. 

The projects funded under Part D are intended to give us some exemplary type 
vocational activity in every state and territory in the Union. Therefore, Congress 
wrote a formula into the Act which stipulates that whatever amount of money is ap- 
propriated will be pro-rated among the states and that grants will be awarded in 
every state and territory. The purpose, then, was to create a bridge between the 
research and development which is done under Part C of the Act and the actual op- 
erations out in the field; Part D is the bridge between the operator in the field 
and the researcher who is working under Part C. 

So it seems to me that this is what Congress expects of all those who work in 
exemplary projects - that they will, in fact, constitute a bridge and bring into 
operation some research-based exemplary projects in every state and territory. 



Working on a State-By-State Basis 

When one looks at the problem of improving education across the nation, he is 
staggered by the fact that there are more than 19,000 separate school districts in 
the United States. A lot of people throw up their hands, saying that it is just 
about impossible to bring about change in that many school districts. However, I 
think if we look at it within the frame of reference of Part D of the Act, the prob- 
lems become much more manageable. Now that Congress has said, ”We want some exem- 
plary activities going on in every state," we can begin to look at the number of 
districts in each state. For example, in the state of Delaware, there are twenty- 
one school districts. If in even one of those school districts, one can bring an 
exemplary vocational education program into operation, it’s within reason to assume 
that it could begin to spread to the other twenty districts in that state. In other 
words, it becomes a manageable problem when viewed on a state-by-state basis. One 
doesn’t have to move all 19,000 school districts at once. Rather, within each state 
there can be established tailor-made exemplary projects that are used as leverage to 
bring out changes that Suit the needs and conditions in that particular state. 

I believe that in general Part D of the Vocational Education Amendments of 1966 
has been a carefully thought out, well conceived piece of legislation which provides 
a good base from which to work. Those concerned with vocational exemplary programs 
have a relatively clear mandate from Congress as to what is expected of them, and 
it obviously behooves them to fulfill these expectations as efficiently as possible 
in order to convince Congress that this sort of effort is worthy of their continu- 
ing attention and support. In this regard, the Office of Education will be needing 
all sorts of feedback from project managers as time goes on in term? of advances 
that they are making, successes they are achieving, and other things that can be 
reported to Congress as evidence of the efficiency with which this job is being 
done . 
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Division of Responsibility Between Project M anagers and the State Office 
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At this stage of the game, it is important to distinguish between the two 
kinds of people who are involved in each exemplary project. First we have the 
local project directors (the people who will actually manage the individual 
projects) and second, we have personnel in state divisions of vocational educa- 
tion. These really are two different kinds of people with different concerns. 

Three major features of the exemplary projects are: 1) hew does one go 

about setting up, bringing into operation and effectively managing it? 2) eval- 
uating the project, and 3) disseminating or diffusing the techniques developed to 
other school districts in the state. It may be useful to divide these three as- 
pects up in relation to the two kinds of people mentioned above. 

Obviously, starting up the project, managing it and conducting it in an ef- 
ficient manner is the responsibility of the local project director. On the other 
hand, dissemination is the primary responsibility of the state office of voca- 
tional education. The local project director really shouldn’t be expected to 
"toot his own horn” and to try to sell his system and techniques around the state. 
In contrast, it is a perfectly valid function for the state office to spread the 
word around to other school districts — to use various means to get other school 
districts to look at and adopt the exemplary program. 

The third aspect, evaluation, is a shared function. The local project di- 
rector has to be concerned with evaluation as a management tool, if far no -other 
reason. He needs continuous feedback from evaluation so that, for example, he 
may discover that certain parts of the operation are not working as they should 
and he should therefore put more manpower into these phases or use a different 
approach. At the same time, evaluation is obviously an important concern of the 
state people because unless they have a rigorous evaluation to assure them that 
this project is good and is doing what they want it to do, they shouldn’t take 
extensive steps to disseminate it. In other words, they want to be sure they have 
a good product before they begin to market it. Needless to say, evaluation is of 
great concern to the Federal group, also, because it has been written into the of- 
ficial regulations that we must have an independent, third-party evaluation of each 
exemplary project. 

I would like to highlight the role of the regional offices of the U. 3. Office 
of Education in connection with exemplary projects. I think that in Part D, more 
than in any other part of the Act, we have relied very heavily on the regional of- 
fices and they have been heavily involved. We've depended or them to get around 
and work with the states for the simple reason that there are so few of us in our 
Branch that it is physically impossible for us to do it all. Had it not been for 
the effective assistance of the regional offices, we wouldn't have been able to 
move as far as we have. We expect to continue this close working relationship 
with them; we hope that they will be involved with us in site visits and periodic 
checks on the projects, «nd that they will continue to be sources of technical 
assistance to project managers. 
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These, then, are the broad expectations of the Act — the sorts of broad 
things that Congress seems to expect from Federal personnel, state officials, 
and local project directors. 



First there is a need to sharpen project objectives and put them into more 
measurable terms. Further, while every project is directed in a general 
sense toward the five broad goals stated by the U. S. Office of Education, there 
is a need in most of the proposals to relate each cluster of specific objectives 
directly to one of these goals. 

The second expectation from project managers is that they will develop an 
evaluation plan. This is not difficult to do if they have a good set of objec- 
tives; that is, if the objectives are properly stated, it is not too difficult to 
determine what criteria would be appropriate for indicating the extent to which 
each objective has been accomplished and then to lay out the details of an evalua- 



A third thing project managers need to give attention to is the development 
of a network analysis charting of the procedure they are going to use . While it 
doesn't have to be PERT (although this may he the best approach), there should be 
some sort of analysis chart which lays out in visual form the steps that are going 
to be taken at least during the first twelve months of the project. 

Charting out the objectives is not very difficult if one has a clear concept 
of 1 ) exactly what his specific objectives are and, 2) what steps he is going to 
take to arrive at each objective. As one begins to chart the procedures out, he 
may find that he has omitted some necessary steps, These omissions vrill show on 
the chart as things which don't link up. 

For those who are not yet familiar with charting techniques, I would recommend 
that they examine a little booklet entitled Program Evaluation and Review Technique : 
Applications in Education . The booklet, which was written by Desmond L. Cook of 
Ohio State University, is available in the ERIC system under the file number ED-015- 
533 and can be purchased on microfiche for fifty cents. 

The charts should address the five broad goals which are built-in features 
of every exemplary project and which stipulate that provisions should be made for 
1) broad occupational orientation for all students at the elementary and secondary 
levels, 2) work experience, cocperative education and similar programs, 3) specific 
training in job-entry skills before leaving school for students not previously 
enrolled in vocational programs, 4) intensive occupational guidance and counseling 
during the last years of school and initial placement of all students upon comple- 
tion of their studies; and 5) carrying on the program with support from regular 
funding sources after a maximum of three years of Federal assistance (U.S.O.E. 

Bureau of Adult, Vocational, and Library Programs, "Highlights of Provisions for 
Exemplary Programs and Projects in Vocational Education." Policy Paper Mo. AVL-V70-1 , 
Washington, P. C. , October 2, 1969). 



What is Expected of Project Managers 



ti'in plan. 
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Charting has many advantages for the project director. For instance, if in 
any one element, he sees that the work is falling behind schedule and that some 
of the other elements are therefore going to be held up, he is then aware of the 
fact that he has to do something to speed up that aspect so that it won t ..ail to 
cut in at the right time and delay progress in the other elements. Charting is 
simply a useful management tool to which all managers ought to devote some time- 
An annual PERT chart may be the best approach. Toward the end of the first year, 
managers could begin preparing the chart for the second twelve-month period. 

A fourth expectation we have of the project director is that he make ">“imum 
use of off-the-shelf material and do a minimum amount of new developmen.. As 
have tried to indicate, this is not a research and development program i we want to 
make the maximum use of the many things on which research and development have al- 
ready been done. Of course, research findings have to be adapted to each local 
situation? Sever, the manager can start with the validated studies that are al- 
ready available and adapt them in his project to the maxtoum extent Possible. Gwen 
the limited funding available (somewhere between and E* ?? 

per year), it lust isn't possible to do extensive development. Instruct.onal mat- 
rial' development costs a lot more money tr.jn is available. So the funds have .o 
be used as operational money for making the projects go. 

For example, Worth Carolina State University's Center for Occupational Educa- 
tion has recently announced that for SlO, within a few days a compul t^f djsearch 
Will be made of the descriptors of all 3S.000 ERIC documents ^aiing wi,h educa 
tional research ani a print-out will be prepared which lists the title and file 
number of every document tha r might be relevant to the inquirer. Just J 

ago/this wou^ have seemed impossible'. Each exemplary project offlgrtogd be 
equipped with a little desk top microfiche reader (which can b p ‘ 

$100-$200). An overview of the kinds and brands of micro ic entitled "A 

on the market can be found in an article by V. D. Tate and R. D. Wolf entitled A 
S^udy of Microfiche Readers and Reader-Printers Currently Manufactured in the 

STfeeatl 

is provided in a “i“ 0 gr a ^ed f Paper .'"^‘lsd ? ree of charge by 

writ ing^to: ERIc'ciearinghouse for the Teaching of English, National council on 

Teaching of English, 503 South Sixth Street, Champaign, Illinois 61 . . . 

„e have already had a search made of all the ERIC documents available as of 
November 1963. and the ERIC specialists pulled cut for f “ ° { £' 
fifty of the most important documents related to each of our broad g 

We got that information out to the field in December of 1969. The listings 

and ED- 

033-25U. 
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The first microfiche (ED-033-255) deals with documents related to our broad 
occuptional awareness goal. It describes projects that have attempted to develop 
materials and techniques for elementary and secondary school youngsters to make 
them aware of the; broad range of occupations in the world of work. It lists the 
titles, the "ED" file numbers and the cost of all the documents that are cited. 
Similarly, each of the other microfiche (ED-033-254, ED-034-062, ED-034-061) 
focuses on another of the broad goals and provides the 'cities, the "ED" file num- 
bers, arrd the cost of selected documents related to the accomplishment of that 
goal. 



It costs about $100 to buy microfiche reproductions of all the documents that 
are cited with "ED" numbers in all four of these basic or "bibliographic" micro- 
fiche. That means that for about $100 a project manager would have everything out 
of the ERIC system that, as of November 1969, was considered of critical importance 
to the kind of work he will be doing in his exemplary project. 

Let us assume that he spends $150 for a moderately priced, desk-top microfiche 
reader, plus the $100 for the microfiche reproductions of significant documents. 

For a total of $250, he will thus have available the whole backlog of related re- 
search and development work, right on his own premises, so that his staff members 
can sit right in their own exemplary project office and review the research and de- 
velopment materials in relation to the project components which they are developing. 
And I do not think $250 is too much to spend to give a project a pipeline into the 
one-half billion dollars worth of research and development results that are stored 
in the ERIC system. 

From time to time, we will try to help update the collection of relevant 
microfiche. It is to be assumed that as ongoing research and development projects 
are completed, additional materials will be flowing continually into the ERIC sys- 
tem; some of these new materials will undoubtedly be useful in connection with work 
on the five broad goals of the exemplary projects. As these new materials accumu- 
late, we will try to mail out a notice to all of our exemplary project directors, 
perhaps every six months or so, listing the titles and the "ED" file numbers of new 
ERIC documents which we think might be useful in the development and operation of 
exemplary projects. This will provide for a continual flow of new research and de- 
velopment materials into the framework of the exemplary projects. I think it is 
incumbent upon project managers to follow through, to take advantage of the ERIC 
system, and to try to keep serving as a bridge between research and development work 
and the operations and are going on in the projects. 

The fifth expectation (and I suppose we are going down now to the more mundane 
levels) is that each project director will submit quarterly reports to the Office 
of Education. The first report will be due ninety days after the starting date, 
with subsequent reports due every three months thereafter. 

Thus, every three months, we will have some sort of feel of how each project 
has moved, how things are going in general, and where the problem areas are. The 



report doesn't have to be elaborate (I don’t think any of us in the Pilot and 
Demonstration Branch want to have to go through a 100. or 200 page report). How- 
ever » it should be comprehensive enough to establish in each project's files and 
in our records an account of what happened during that quarter (e.g., what pro- 
gress was made? Did the project fall behind schedule? If so, what are the prob- 
lems? Are there any kinds of things that the U.S.O.E. Washington Office or U.S.O.E. 
Regional Office or the State Division of Vocational Education might to do help 
overcome these problems?) Those are the kinds of things which should be put in the 
quarterly report, which, in essence, will be our main way of keeping in touch with 
what's going on within each project. 

Beyond that, the requirements call for an interim report once a year. (In 
other words, three quarterly reports will be submitted and then, at the end of the 
fourth quarter, there will be an annual interim report prepared which will summarize 
for us what has happened during the whole twelve-month period.) In addition, one 
section of the annual interim report will be the first annual evaluation report 
which the outside, third party contractor has produced. 

Once we have that annual interim report in hand, we are then expected to make 
a site visit. We will expect to join forces with our U.S.O.E. Regional Office 
personnel and with someone from the State Division of Vocational Education and go 
right out to the site of the project to look at the interim report and the evalua- 
tion report, compare them with what's going on, and talk with the project people 
and with students and parents. At that point, a decision has to be made about 
the second year of funding x Obviously, if a project is going sour and failing to 
make progress, there is no use putting good money after the bad. It is better to 
shut it off and start another exemplary project somewhere else in that state. On 
the other hand, if things are going along smoothly and good progress is being 
made, we want to put the second and third years of funding in and keep the project 
moving . 



Budgetary 



In addition, at this time we will also expect to examine the budget for the 
second year and the PERT chart (or equivalent), showing the precise steps to be 
followed during the next twelve months. Toward the end of the first twelvd months, 
the project manager ought to be able to project ahead fairly accurately in terms of 
what his second-year requirements are going to be. He may decide that he needs more 
money in "Supplies" and less money in "Personnel," or that he needs more money for 
travel and less money for something else, further his understanding of. the require- 
ments will change over the course of the first year. During the site visit, we will 
take these changes, as well as what was accomplished during the first year, into ac- 
count in projecting ahead toward the second twelve months. 

The budget which is attached to the contract is rigid once the contract is 
signed by the project manager's institution and the U.S.O.E. Contract Office. If 
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the project manager wants to change, he has to write to Washington and give a 
rationale for the proposed changes (e.g., he might say something like: "We find 
that we had too much money in travel: on the other hand, we have found some exci- 
ting instructional materials which we want to use in our program. So, v;e want to 
shift some money out of the "Travel" item into the "Instructional Materials" item) 
If the request is reasonable, I think it is fairly easy to have the budget amended 
But the manager must be sure to go through the step of submitting a written amend- 
ment to the budget; he should not just start unilaterally spending in a new fund- 
ing pattern without a prior written clearance from the U.S.O.E. Contract Officer. 

The second year budget is still flexible, however; the initial contract is 
signed with only the first year budget detailed and firm. When we are ready to 
fund the second year, the project director will have a chance to readjust the bud- 
get in any way he thinks necessary and negotiate on that basis with the Contract 
Officer. 



It can hard.l j be overemphasized that any changes in the scope of work in the 
project or in the budget require official amendment to the contract. Managers 
should send a written request if they want to make any substantial change in the 
scope of work, and get it approved as an amendment to the contract, regardless of 
what has been agreed to by us or others. If this is not done, an auditor can later 
disallow any expenses incurred in the change. 

As a matter of fact, there are regulations that say that all the reports and 
rscorcs of a project have to be kept for at least five years after the terminal 
date of the project. At any time during that period, the Federal auditors can come 
in and check the approved budget against all the receipts, vouchers, etc., in an 
attempt to find discrepancies. Thus, it is very important for the manager to keep 
all expenses related to particular line items in the budget so that the books will 
protect him during an audit. 

Finally, of course, there are also allowable expenses spelled out in the Fed- 
eral Regulations. Project managers will soon receive a copy of the final regula- 
tions, which were revised for printing in July (Refer to the Federal Register , Vol. 
35, No. 143: Friday, July 24, 1970; Part II; Sub-parts A, C, and E). Until then, 
they should refer to Appendix B of the preliminary manual which contains a draft of 
the Federal Regulations for Part D projects ( Manual : Instructions and Procedures ; 
Exemplary Programs and Projects in Vocational Education. November 1, 1969). See 
page 22 of the manual for "allowable costs." 

Managers are cautioned not to spend money for anything that is not allowable. 
For example, the construction of buildings is not an allowable cost. 

In concluding, I believe that the exemplary projects in vocational-technical 
education can occupy a key part in the continuum of educational innovation. If 
carefully developed and managed, they will serve as a vital mechanism for accom- 
plishing the translation of research and development work into improved operations 
in the field. 
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Dr. Howard H. Me Faun 
Director, Division #3, HumRRO 



The purpose of this conference, as I see it, is to bring together the rele- 
vant parties involved in exemplary programs for: 1 ) a greater understanding of 
the reasons for emphasis on these projects; 2) a means of guidance for the develop- 
ment of a rationale as to what is expected by all parties involved in the projects; 
3) suggestions of possible approaches to be taken in project development along 
with pointing out some of the pitfalls of the same; and 4) an opportunity for the 
people to become better acquainted and aware of the fact that they do indeed share 
a common gcal. Let me comment briefly on the emphasis of this week's program as 
T heard it described. 

It goes back, I guess, to a mandate From the public, the president, and the 
Congress, for education to produce. And as one gentleman stated it, "Paint or 
get off the ladder." I think this describes very well tho position of our public. 

There also has been a formal recognition of the problem which was given by 
the Associate Commissioner, Dr. Hurdvick, when he stated, "In the administration 
and levels of efficiency in most state vocational staffs, you will find incompe- 
tencies, fumbling, and few services rendered; further, their vocational education 
programs cannot be defined and the deficiencies cannot be measured." Then I think, 
there has been, very importantly, a formal recognition by Congress of the impor- 
tance of vocational education. This, to me, reflects back on the fact that the 
public sees the great role vocational education can play. It's a role I happen to 
feel very strongly about. Because of Sputnik, people got all concerned about the 
academic programs and they forgot all about the vocational part of education. Now 
people are again becoming aware of the fact that there is this great potential in 
vocational education. This is a great responsibility and this is a fact that came 
through a great deal in the comments from many of the early speakers. In Title I 
it was difficult to see any change and I think this accounts for the emphasis here 
on the aspect of outside evaluation. You are going to have to point very carefully 
to the processes you have used, and the aspect of dissemination comes in at this 
point. I suspect there were some very good things done under Title I, but they 
go*- lost. I believe this must have been the reason for the new emphasis at the 
Federal level at which it was decided that we should take these Part D funds and 
make sure they are applied in a systematic way, rather than saying, "Here is 
some money, now do your best." 

In spite of mandated criteria, great freedom has been given to each project, 
and I think there is an expectation of a wide variety of approaches and solutions 
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in each case. This was further elaborated when various examples of programs were 
pointed out. I think it is the intent for you to show that there are a lot of ways 
to solve the problem. Nevertheless, it is a common problem that you are to solve; 
you are not to try to solve a variety of problems. The great emphasis all the way 
through is upon getting your objective stated explicitly so that it will be clear 
what problem you are trying to solve. It's somewhat the old story of M if you 
don't know where you are going, it doesn't matter much what route you take." I 
think in the past, many of us have been guilty of finding that there were some very 
interesting scenic routes, but we didn’t quite know where we were going. 

There was another expectation stated. It was that you should be truly innova- 
tive. You are to make waves and not represent the status quo. I see your job as 
that of an engineer. You are not researchers, you are not developers, you are en- 
gineers. You are applying technology to the solution of a problem. Now obviously 
you have to tailor when you start engineering. I think you have to very carefully 
state your constraints and I think you have to be extremely sensitive to what modi- 
fications have been made by you as an engineer. 

I think there is another point which came through. It is expected that you 
will solve the problem and even if you don't reach the optimal goal, you can and 
will reach a satisfactory solution. A great deal is resting on you people. I 
am convinced there is a great deal of educational technology which can be applied 
to your projects. Obviously you people are sensitive to this technology. There 
was talk about role differentiation, about the use of the systems approach, etc. 

I do believe the educational technology available has sorely lacked application, 
and here is an opportunity to do it. 

It also is expected that you are to be able to state explicitly the problem, 
and the process employed in attempting to solve the problem, ,\nd how closely you 
come to reaching your desired objectives. A considerable amount of attention was 
given to how you might evaluate. Early in the session there was an idea that you 
might have a common evaluation procedure. This would make sense only if you were 
all working on the same objective. You are working on the same goal, but not on 
the same objective. I think when you start your engineering, you will have to tie 
your evaluation very closely to that engineering. I would personally feel that 
you would have to conduct specific evaluations as opposed to a general one . In 
regard to other aspects of the evaluation, I would encourage you to use multiple 
criteria. Don't go for broke on particular criteria. Another thing I would en- 
courage is that when you are looking at your evaluation initially, don't be too 
concerned at first glance if you don't know how to measure it. Spend a little time 
trying to be innovative in this sense. The idea of the outside evaluator goes way 
back to what the impetus is for this whole evaluation activity. Can you do your 
own evaluation? Well, let me ignore the methodological problems of people trying 
to evaluate themselves. We can all try to evaluate ourselves but there can be 
problems. The public desires to have an outside evaluation so an outside individual 
can say to the public, "Yes, they really did succeed in this manner." It becomes 
much more believable. This is one of the reasons why the oucside evaluation is more 
necessary. 

This is also where accountability comes into focus. It is not just accounta- 
bility to the profession, it is also accountability to the public. 
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Everyone agreed with the idea of using some type of systems approach. Bob 
Barnes gave a very good flow chart; it was excellent because it had been derived 
empirically and because it gave you a lot of decision points. 

The administration of your project implies that you have stated explicitly 
your goals. It also implies you then have a system, perhaps a resource allocation 
system. I think this is going to be one of the real problems you will have in 
project management. There is so much to be done; where do you spend your money? 

This is all the more reason for stating what is going on and track it very care- 
fully from inside your own management system. I think everyone realizes that 
evaluation is not going to occur outside the system. As project managers, you 
have to track very carefully what is going on and you have to have continuous 
evaluation, and that outside evaluation is a supportive, corollary, and cooperative 
evaluation. 

Everyone has told you to define the problem using the systems approach. There 
is quite a bit of literature on this and there are quite a few documents that have 
been written on how to state objectives. A whole series of people have written on 
this and I would encourage looking at these documents , because a systems approach 
does take a little practice and since other people have gone through it, there is 
no sense in your having to start from scratch. I think there has been a fair 
amount of good material written on the development of instructional systems. 

Also, keep in mind the fact that the Army, the Air Force, and the Navy have, 
for twenty years, been producing a great deal of research and there is a lot of 
package stuff they have developed which you can use. They have many of these pro- 
grams and some of them have been tried out and proved to be successful* This is a 
great source for information and I would encourage your use of this material. 

Let me talk a little bit about what I heard people saying you ought to be 
doing generally. They say you ought to be talking about the change in the role of 
teachers where the teacher should not be the imparter of information but the teacher 
should take on the role of diagnostician, maintained, and programmer in some form. 

As you try to put into effect the programs, the biggest problem you are going to 
have is to get adequate change and acceptance on the part of the people you are try- 
ing to have put in the program. Top management is easy to talk to, but when you get 
down to the level of the operators, you have a great deal of trouble getting them to 
change their behavior and I would encourage some systematic quality control to as- 
sure that what you think is happening is really happening. Otherwise you will get 
all kinds of verbal agreement and still get no change in behavior. Also, in my ex- 
perience, the vocational educator has been viewed as a second-class citizen within 
many of the school systems. We still need quite a bit of selling to overcome some 
of these concepts. As someone said, ’’Vocational education is always for someone 
else’s child.* 1 

There seems to be a great deal of emphasis here on ” individualization," which 
is a good catch word. It is, nevertheless, very difficult to attain. As soon as 
you talk about individualization, you are talking about management problems. This 
is why we have stayed with the extremely efficient system we have, even though it is 
not very effective. 
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For the last three or four years, we have been working on the problem of "so- 
called 11 lew aptitude individuals . He talk about the young fellow who failed in 
the school system and then was taken into the Armed Forces. He had been told so 
many times that he was a "successful failure" that he believed it. I think one 
of the things you are going to find out is that when you start dealing with these 
people, this whole problem of self-concept is going to be a very big concern. This 
deals with attitudes and they are almost impossible to measure. If you forget 
that, you are in trouble- He have found this to be extremely important. He also 
have found that with the low aptitude person, when you get rid of this subject mat- 
ter approach and use a fact approach (where there is a meaningful, activity which he 
can see), then he does learn and it turns him on to further learning. He have also 
found that it takes this fellow about fc'ar or five times as long to learn, so this 
means you are going to have to set up your program to allow for variable time ele- 
ments. Our data also shows that after he has learned a task, he will retain it as 
long as anyone wise. He retains it as well as if he had mastered it on his own. 

So please, go for a mastery concept and do not allow this youngster to move on to 
another level before he has mastered the enabling level. This is why youngsters 
who have spent ten years in our school system are reading at the fourth grade level. 
We have had the experience of a person finding a written page so adverse that they 
cannot look at it. That says something about our school system and the problems you 
will have to face. It has to go towards simulation, towards a task approach; we must 
get away from subject matter. 

He have also found that the training method approach for the low ability indi- 
vidual is critical as to whether or not he learns . The guy down at the bottom needs 
instruction; he needs support; he needs to have a system of success. And if you pro- 
vide this, he does turn on, and he does learn. 

One of the problems you have in achieving this is the teacher-student ratio. 

How do you achieve this when you have one teacher and thirty students? A technique 
we have found very valuable is the buddy and peer system of instruction. I would 
encourage your consideration of this technique as long as you have good quality 
control. By this I mean you must ensure that the one peer who is going to play 
the role of the teacher has in fact learned and mastered the material. He found in 
som<>. studies that peer instructors talk to people, and then the people they talked 
to talk to other people. The result is that eventually everyone has these skills. 

He also found out that when we used an outside evaluation, the quality of instruc- 
tion did not go down; mastery maintained itself. Hhat did go down was the time 
needed to teach a concept. They threw out all the superfluous material and they 
knew what was important . 

This gets to another point I want to make on the statement of objectives: 

Hhen are we going to consider getting the student involved in determining objec- 
tives? If we were really talking about innovation and change, then we must con- 
sider the student’s objectives and goals, I think they tend to have fairly realis- 
tic ones . 

Hhen you talked about dissemination, it became clear that you arc agents for 
change and that is what exemplary programs are all about. It is important in the 
dissemination process that you record what you did, how you did it, and how success- 
ful you were in the process. This is another reason for the emphasis being placed on 
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record keeping, on evaluation. However, it is absuX'd to think that you have the 
primary responsibility as agents for change. The primary responsibility is at the 
Federal level and they have initiated it by starting these projects. If you have 
adeauate information and are dedicated to your project, I refuse to believe that 
you can fail completely. It may not meet your ultimate objectives, but you should 
come close. 

The next stage of dissemination involves the movement of information from you 
to others throughout the country. That's why you have to be careful to record 
everything and then forward it up to the state level and from there, up to the Fed- 
eral level. They also have a role, a responsibility, in this process of dissemina- 
tion. 



I also think a lot of role definition has occurred, and there have been quite 
a few suggestions on possible approaches to this problem. The systems approach, 
for one, has been repeatedly suggested. You know what this means. You must de- 
fine your problem in very explicit terms so that you can then set up measuring 
instruments. It is not so profound. Most of the time I notice that in develop- 
ment work, we jump into the middle of the problem before it is defined I would 
encourage you to spend a good amount of your time early in this process explicitly 
stating ar much as you can about your objectives, your approaches, and your 
resource allocations. 

One thing not mentioned but which was alluded to was the question regarding 
some form of contingency management. This is an extremely powerful technique and 
it doesn't always call for money to be expended. All it says is that you find out 
what is important to the success of the program and then you set up the contingen- 
cies . 

I think this Institute has been an excellent opportunity for people to become 
very much better acquainted. There is a desire for greater communication between 
project directors and obviously this is necessary. There is also the danger that 
one or two of you are innovators and the rest of you are copiers. I don’t think 
that was the intent of the legislation. Rather the intent here is to help you 
discover a broad variety of ways to solve the problem. I would therefore encourage 
you not to limit your communi cat ions to the other project directors but also to 
use ERIC as well as other resources . 

In closing, I would say to you as you go down this exemplary banister, I hope 
that all the splinters are pointing downwards fdr each of you along the way. 
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